2024-03-29T10:29:59.837 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-29T10:29:59.837 DEBUG:teuthology.run:Teuthology command: teuthology --archive /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622648 --owner scheduled_amathuri@teuthology --name amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi --verbose --description rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zstd rados tasks/rados_api_tests validater/lockdep} -- /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622648/orig.config.yaml 2024-03-29T10:29:59.873 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-29T10:29:59.958 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622648 branch: main description: rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zstd rados tasks/rados_api_tests validater/lockdep} email: null first_in_suite: false job_id: '7622648' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: main ceph: conf: client: debug monc: 20 debug ms: 1 debug objecter: 20 debug rados: 20 global: lockdep: true mon client directed command retry: 5 mon election default strategy: 1 ms bind msgr2: false ms inject socket failures: 5000 ms type: async mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon scrub interval: 300 mon warn on pool no app: false paxos service trim min: 10 osd: bluestore block size: 96636764160 bluestore compression algorithm: zstd bluestore compression mode: aggressive bluestore fsck on mount: true bluestore zero block detection: true debug bluefs: 20 debug bluestore: 20 debug monc: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd class default list: '*' osd class load list: '*' osd client watch timeout: 120 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random osd shutdown pgref assert: true flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME - reached quota - overall HEALTH_ - \(CACHE_POOL_NO_HIT_SET\) - \(POOL_FULL\) - \(SMALLER_PGP_NUM\) - \(SLOW_OPS\) - \(CACHE_POOL_NEAR_FULL\) - \(POOL_APP_NOT_ENABLED\) - \(PG_AVAILABILITY\) - \(OBJECT_MISPLACED\) - slow request mon_bind_msgr2: false sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: main sha1: 1fb5d10608dccee73edd6ef8a2c7b804a097c324 owner: scheduled_amathuri@teuthology priority: 150 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - node-exporter.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - node-exporter.b seed: 8152 sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 sleep_before_teardown: 0 subset: 99/1000 suite: rados:verify suite_branch: main suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 1fb5d10608dccee73edd6ef8a2c7b804a097c324 targets: smithi088.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJC0hr90S7oLfAoelGBHu4/0U2rkf61hfisamgYRLn7TC82DQRaG4HpL3kXnRcujlwQVB8jMOFZCfEWpvvZDsdI= smithi196.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOmc8TxTRpU1Lg3jXUaU3i4KUHAnuTYSGL8ZhlCPHEUiKLNgxAitJvjwBVcGhpvmK8w1uOgNB2wJREwIlBz5vvs= tasks: - install: null - ceph: null - workunit: clients: client.0: - rados/test.sh env: ALLOW_TIMEOUTS: '1' timeout: 6h teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-26_05:45:44 tube: smithi user: amathuri verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-29T10:29:59.958 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa; will attempt to use it 2024-03-29T10:29:59.959 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks 2024-03-29T10:29:59.959 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-29T10:29:59.961 INFO:teuthology.task.internal:Checking packages... 2024-03-29T10:29:59.987 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '5dcb190da270440ef141d088eb27a9fdb8afc5f2' 2024-03-29T10:29:59.987 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-29T10:29:59.987 INFO:teuthology.packaging:ref: None 2024-03-29T10:29:59.987 INFO:teuthology.packaging:tag: None 2024-03-29T10:29:59.987 INFO:teuthology.packaging:branch: main 2024-03-29T10:29:59.987 INFO:teuthology.packaging:sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-29T10:29:59.987 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=main 2024-03-29T10:30:00.375 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2530.gb74786fc 2024-03-29T10:30:00.376 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-29T10:30:00.383 INFO:teuthology.task.internal:no buildpackages task found 2024-03-29T10:30:00.383 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-29T10:30:00.431 INFO:teuthology.task.internal:Saving configuration 2024-03-29T10:30:00.445 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-29T10:30:00.457 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-29T10:30:00.484 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi088.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622648', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-29 10:24:34.037213', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJC0hr90S7oLfAoelGBHu4/0U2rkf61hfisamgYRLn7TC82DQRaG4HpL3kXnRcujlwQVB8jMOFZCfEWpvvZDsdI='} 2024-03-29T10:30:00.511 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi196.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622648', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-29 10:24:34.035537', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOmc8TxTRpU1Lg3jXUaU3i4KUHAnuTYSGL8ZhlCPHEUiKLNgxAitJvjwBVcGhpvmK8w1uOgNB2wJREwIlBz5vvs='} 2024-03-29T10:30:00.511 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-29T10:30:00.518 INFO:teuthology.task.internal:roles: ubuntu@smithi088.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2024-03-29T10:30:00.519 INFO:teuthology.task.internal:roles: ubuntu@smithi196.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2024-03-29T10:30:00.519 INFO:teuthology.run_tasks:Running task console_log... 2024-03-29T10:30:00.618 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f8c95f2c0d0>, signals=[15]) 2024-03-29T10:30:00.619 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-29T10:30:00.626 INFO:teuthology.task.internal:Opening connections... 2024-03-29T10:30:00.626 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi088.front.sepia.ceph.com 2024-03-29T10:30:00.628 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi088.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T10:30:00.721 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi196.front.sepia.ceph.com 2024-03-29T10:30:00.722 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi196.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T10:30:00.813 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-29T10:30:00.820 DEBUG:teuthology.orchestra.run.smithi088:> uname -m 2024-03-29T10:30:00.837 INFO:teuthology.orchestra.run.smithi088.stdout:x86_64 2024-03-29T10:30:00.838 DEBUG:teuthology.orchestra.run.smithi088:> cat /etc/os-release 2024-03-29T10:30:00.894 INFO:teuthology.orchestra.run.smithi088.stdout:NAME="CentOS Stream" 2024-03-29T10:30:00.894 INFO:teuthology.orchestra.run.smithi088.stdout:VERSION="9" 2024-03-29T10:30:00.894 INFO:teuthology.orchestra.run.smithi088.stdout:ID="centos" 2024-03-29T10:30:00.894 INFO:teuthology.orchestra.run.smithi088.stdout:ID_LIKE="rhel fedora" 2024-03-29T10:30:00.894 INFO:teuthology.orchestra.run.smithi088.stdout:VERSION_ID="9" 2024-03-29T10:30:00.894 INFO:teuthology.orchestra.run.smithi088.stdout:PLATFORM_ID="platform:el9" 2024-03-29T10:30:00.894 INFO:teuthology.orchestra.run.smithi088.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-29T10:30:00.894 INFO:teuthology.orchestra.run.smithi088.stdout:ANSI_COLOR="0;31" 2024-03-29T10:30:00.895 INFO:teuthology.orchestra.run.smithi088.stdout:LOGO="fedora-logo-icon" 2024-03-29T10:30:00.895 INFO:teuthology.orchestra.run.smithi088.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-29T10:30:00.895 INFO:teuthology.orchestra.run.smithi088.stdout:HOME_URL="https://centos.org/" 2024-03-29T10:30:00.895 INFO:teuthology.orchestra.run.smithi088.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-29T10:30:00.895 INFO:teuthology.orchestra.run.smithi088.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-29T10:30:00.895 INFO:teuthology.orchestra.run.smithi088.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-29T10:30:00.895 INFO:teuthology.lock.ops:Updating smithi088.front.sepia.ceph.com on lock server 2024-03-29T10:30:00.918 DEBUG:teuthology.orchestra.run.smithi196:> uname -m 2024-03-29T10:30:00.935 INFO:teuthology.orchestra.run.smithi196.stdout:x86_64 2024-03-29T10:30:00.935 DEBUG:teuthology.orchestra.run.smithi196:> cat /etc/os-release 2024-03-29T10:30:00.992 INFO:teuthology.orchestra.run.smithi196.stdout:NAME="CentOS Stream" 2024-03-29T10:30:00.992 INFO:teuthology.orchestra.run.smithi196.stdout:VERSION="9" 2024-03-29T10:30:00.992 INFO:teuthology.orchestra.run.smithi196.stdout:ID="centos" 2024-03-29T10:30:00.992 INFO:teuthology.orchestra.run.smithi196.stdout:ID_LIKE="rhel fedora" 2024-03-29T10:30:00.992 INFO:teuthology.orchestra.run.smithi196.stdout:VERSION_ID="9" 2024-03-29T10:30:00.992 INFO:teuthology.orchestra.run.smithi196.stdout:PLATFORM_ID="platform:el9" 2024-03-29T10:30:00.992 INFO:teuthology.orchestra.run.smithi196.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-29T10:30:00.992 INFO:teuthology.orchestra.run.smithi196.stdout:ANSI_COLOR="0;31" 2024-03-29T10:30:00.993 INFO:teuthology.orchestra.run.smithi196.stdout:LOGO="fedora-logo-icon" 2024-03-29T10:30:00.993 INFO:teuthology.orchestra.run.smithi196.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-29T10:30:00.993 INFO:teuthology.orchestra.run.smithi196.stdout:HOME_URL="https://centos.org/" 2024-03-29T10:30:00.993 INFO:teuthology.orchestra.run.smithi196.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-29T10:30:00.993 INFO:teuthology.orchestra.run.smithi196.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-29T10:30:00.993 INFO:teuthology.orchestra.run.smithi196.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-29T10:30:00.993 INFO:teuthology.lock.ops:Updating smithi196.front.sepia.ceph.com on lock server 2024-03-29T10:30:01.018 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-29T10:30:01.027 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-29T10:30:01.049 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-29T10:30:01.049 DEBUG:teuthology.orchestra.run.smithi088:> test '!' -e /home/ubuntu/cephtest 2024-03-29T10:30:01.052 DEBUG:teuthology.orchestra.run.smithi196:> test '!' -e /home/ubuntu/cephtest 2024-03-29T10:30:01.067 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-29T10:30:01.074 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-29T10:30:01.074 DEBUG:teuthology.orchestra.run.smithi088:> test -z $(ls -A /var/lib/ceph) 2024-03-29T10:30:01.108 DEBUG:teuthology.orchestra.run.smithi196:> test -z $(ls -A /var/lib/ceph) 2024-03-29T10:30:01.177 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-29T10:30:01.235 INFO:teuthology.run_tasks:Running task kernel... 2024-03-29T10:30:01.250 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-03-29T10:30:01.250 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-03-29T10:30:01.251 DEBUG:teuthology.orchestra.run.smithi088:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-29T10:30:01.251 DEBUG:teuthology.orchestra.run.smithi196:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-29T10:30:01.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T10:30:01.267 DEBUG:teuthology.orchestra.run.smithi088:> uname -r 2024-03-29T10:30:01.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T10:30:01.268 DEBUG:teuthology.orchestra.run.smithi196:> uname -r 2024-03-29T10:30:01.324 INFO:teuthology.orchestra.run.smithi088.stdout:5.14.0-361.el9.x86_64 2024-03-29T10:30:01.324 INFO:teuthology.task.kernel:Running kernel on smithi088: 5.14.0-361.el9.x86_64 2024-03-29T10:30:01.324 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum install -y kernel 2024-03-29T10:30:01.325 INFO:teuthology.orchestra.run.smithi196.stdout:5.14.0-361.el9.x86_64 2024-03-29T10:30:01.325 INFO:teuthology.task.kernel:Running kernel on smithi196: 5.14.0-361.el9.x86_64 2024-03-29T10:30:01.325 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum install -y kernel 2024-03-29T10:30:03.423 INFO:teuthology.orchestra.run.smithi088.stdout:CentOS-9 - AppStream 15 MB/s | 6.7 MB 00:00 2024-03-29T10:30:03.492 INFO:teuthology.orchestra.run.smithi196.stdout:CentOS-9 - AppStream 13 MB/s | 6.7 MB 00:00 2024-03-29T10:30:05.057 INFO:teuthology.orchestra.run.smithi088.stdout:CentOS-9 - BaseOS 3.8 MB/s | 2.0 MB 00:00 2024-03-29T10:30:05.243 INFO:teuthology.orchestra.run.smithi196.stdout:CentOS-9 - BaseOS 3.2 MB/s | 2.0 MB 00:00 2024-03-29T10:30:06.449 INFO:teuthology.orchestra.run.smithi196.stdout:CentOS Stream 9 - BaseOS 9.3 MB/s | 8.0 MB 00:00 2024-03-29T10:30:08.494 INFO:teuthology.orchestra.run.smithi196.stdout:CentOS Stream 9 - AppStream 19 MB/s | 19 MB 00:00 2024-03-29T10:30:14.279 INFO:teuthology.orchestra.run.smithi088.stdout:CentOS Stream 9 - BaseOS 925 kB/s | 8.0 MB 00:08 2024-03-29T10:30:14.583 INFO:teuthology.orchestra.run.smithi196.stdout:CentOS Stream 9 - CRB 2.4 MB/s | 6.1 MB 00:02 2024-03-29T10:30:16.159 INFO:teuthology.orchestra.run.smithi196.stdout:CentOS Stream 9 - Extras packages 36 kB/s | 16 kB 00:00 2024-03-29T10:30:17.469 INFO:teuthology.orchestra.run.smithi196.stdout:Extra Packages for Enterprise Linux 17 MB/s | 21 MB 00:01 2024-03-29T10:30:17.988 INFO:teuthology.orchestra.run.smithi088.stdout:CentOS Stream 9 - AppStream 7.2 MB/s | 19 MB 00:02 2024-03-29T10:30:21.980 INFO:teuthology.orchestra.run.smithi088.stdout:CentOS Stream 9 - CRB 13 MB/s | 6.1 MB 00:00 2024-03-29T10:30:22.477 INFO:teuthology.orchestra.run.smithi196.stdout:lab-extras 12 kB/s | 1.7 kB 00:00 2024-03-29T10:30:23.389 INFO:teuthology.orchestra.run.smithi088.stdout:CentOS Stream 9 - Extras packages 56 kB/s | 16 kB 00:00 2024-03-29T10:30:24.718 INFO:teuthology.orchestra.run.smithi088.stdout:Extra Packages for Enterprise Linux 17 MB/s | 21 MB 00:01 2024-03-29T10:30:24.733 INFO:teuthology.orchestra.run.smithi196.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-29T10:30:24.733 INFO:teuthology.orchestra.run.smithi196.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-29T10:30:24.828 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-03-29T10:30:24.838 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-29T10:30:24.839 INFO:teuthology.orchestra.run.smithi196.stdout: Package Arch Version Repository Size 2024-03-29T10:30:24.839 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-29T10:30:24.839 INFO:teuthology.orchestra.run.smithi196.stdout:Installing: 2024-03-29T10:30:24.839 INFO:teuthology.orchestra.run.smithi196.stdout: kernel x86_64 5.14.0-434.el9 CentOS-BaseOS 192 k 2024-03-29T10:30:24.839 INFO:teuthology.orchestra.run.smithi196.stdout:Installing dependencies: 2024-03-29T10:30:24.839 INFO:teuthology.orchestra.run.smithi196.stdout: kernel-core x86_64 5.14.0-434.el9 CentOS-BaseOS 15 M 2024-03-29T10:30:24.839 INFO:teuthology.orchestra.run.smithi196.stdout: kernel-modules x86_64 5.14.0-434.el9 CentOS-BaseOS 34 M 2024-03-29T10:30:24.839 INFO:teuthology.orchestra.run.smithi196.stdout: kernel-modules-core x86_64 5.14.0-434.el9 CentOS-BaseOS 28 M 2024-03-29T10:30:24.839 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-29T10:30:24.839 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction Summary 2024-03-29T10:30:24.839 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-29T10:30:24.839 INFO:teuthology.orchestra.run.smithi196.stdout:Install 4 Packages 2024-03-29T10:30:24.840 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-29T10:30:24.840 INFO:teuthology.orchestra.run.smithi196.stdout:Total download size: 76 M 2024-03-29T10:30:24.840 INFO:teuthology.orchestra.run.smithi196.stdout:Installed size: 123 M 2024-03-29T10:30:24.840 INFO:teuthology.orchestra.run.smithi196.stdout:Downloading Packages: 2024-03-29T10:30:24.986 INFO:teuthology.orchestra.run.smithi196.stdout:(1/4): kernel-5.14.0-434.el9.x86_64.rpm 1.3 MB/s | 192 kB 00:00 2024-03-29T10:30:25.503 INFO:teuthology.orchestra.run.smithi196.stdout:(2/4): kernel-core-5.14.0-434.el9.x86_64.rpm 23 MB/s | 15 MB 00:00 2024-03-29T10:30:26.037 INFO:teuthology.orchestra.run.smithi196.stdout:(3/4): kernel-modules-5.14.0-434.el9.x86_64.rpm 28 MB/s | 34 MB 00:01 2024-03-29T10:30:26.379 INFO:teuthology.orchestra.run.smithi196.stdout:(4/4): kernel-modules-core-5.14.0-434.el9.x86_6 20 MB/s | 28 MB 00:01 2024-03-29T10:30:26.381 INFO:teuthology.orchestra.run.smithi196.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:30:26.382 INFO:teuthology.orchestra.run.smithi196.stdout:Total 50 MB/s | 76 MB 00:01 2024-03-29T10:30:26.382 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction check 2024-03-29T10:30:26.752 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction check succeeded. 2024-03-29T10:30:26.752 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction test 2024-03-29T10:30:27.600 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction test succeeded. 2024-03-29T10:30:27.600 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction 2024-03-29T10:30:29.648 INFO:teuthology.orchestra.run.smithi088.stdout:lab-extras 27 kB/s | 1.7 kB 00:00 2024-03-29T10:30:29.916 INFO:teuthology.orchestra.run.smithi196.stdout: Preparing : 1/1 2024-03-29T10:30:30.755 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : kernel-modules-core-5.14.0-434.el9.x86_64 1/4 2024-03-29T10:30:30.853 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T10:30:31.913 INFO:teuthology.orchestra.run.smithi088.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-29T10:30:31.913 INFO:teuthology.orchestra.run.smithi088.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-29T10:30:31.996 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2024-03-29T10:30:32.006 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-03-29T10:30:32.006 INFO:teuthology.orchestra.run.smithi088.stdout: Package Arch Version Repository Size 2024-03-29T10:30:32.006 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-03-29T10:30:32.006 INFO:teuthology.orchestra.run.smithi088.stdout:Installing: 2024-03-29T10:30:32.006 INFO:teuthology.orchestra.run.smithi088.stdout: kernel x86_64 5.14.0-434.el9 CentOS-BaseOS 192 k 2024-03-29T10:30:32.006 INFO:teuthology.orchestra.run.smithi088.stdout:Installing dependencies: 2024-03-29T10:30:32.006 INFO:teuthology.orchestra.run.smithi088.stdout: kernel-core x86_64 5.14.0-434.el9 CentOS-BaseOS 15 M 2024-03-29T10:30:32.007 INFO:teuthology.orchestra.run.smithi088.stdout: kernel-modules x86_64 5.14.0-434.el9 CentOS-BaseOS 34 M 2024-03-29T10:30:32.007 INFO:teuthology.orchestra.run.smithi088.stdout: kernel-modules-core x86_64 5.14.0-434.el9 CentOS-BaseOS 28 M 2024-03-29T10:30:32.007 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-03-29T10:30:32.007 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction Summary 2024-03-29T10:30:32.007 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-03-29T10:30:32.007 INFO:teuthology.orchestra.run.smithi088.stdout:Install 4 Packages 2024-03-29T10:30:32.007 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-03-29T10:30:32.007 INFO:teuthology.orchestra.run.smithi088.stdout:Total download size: 76 M 2024-03-29T10:30:32.007 INFO:teuthology.orchestra.run.smithi088.stdout:Installed size: 123 M 2024-03-29T10:30:32.007 INFO:teuthology.orchestra.run.smithi088.stdout:Downloading Packages: 2024-03-29T10:30:32.149 INFO:teuthology.orchestra.run.smithi088.stdout:(1/4): kernel-5.14.0-434.el9.x86_64.rpm 1.3 MB/s | 192 kB 00:00 2024-03-29T10:30:32.716 INFO:teuthology.orchestra.run.smithi088.stdout:(2/4): kernel-core-5.14.0-434.el9.x86_64.rpm 21 MB/s | 15 MB 00:00 2024-03-29T10:30:33.250 INFO:teuthology.orchestra.run.smithi088.stdout:(3/4): kernel-modules-5.14.0-434.el9.x86_64.rpm 27 MB/s | 34 MB 00:01 2024-03-29T10:30:33.600 INFO:teuthology.orchestra.run.smithi088.stdout:(4/4): kernel-modules-core-5.14.0-434.el9.x86_6 19 MB/s | 28 MB 00:01 2024-03-29T10:30:33.603 INFO:teuthology.orchestra.run.smithi088.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:30:33.603 INFO:teuthology.orchestra.run.smithi088.stdout:Total 48 MB/s | 76 MB 00:01 2024-03-29T10:30:33.603 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction check 2024-03-29T10:30:33.609 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T10:30:33.739 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T10:30:33.977 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction check succeeded. 2024-03-29T10:30:33.977 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction test 2024-03-29T10:30:34.813 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction test succeeded. 2024-03-29T10:30:34.813 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction 2024-03-29T10:30:37.072 INFO:teuthology.orchestra.run.smithi088.stdout: Preparing : 1/1 2024-03-29T10:30:37.786 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : kernel-modules-core-5.14.0-434.el9.x86_64 1/4 2024-03-29T10:30:37.866 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T10:30:38.452 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T10:30:38.487 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : kernel-5.14.0-434.el9.x86_64 4/4 2024-03-29T10:30:40.636 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T10:30:40.777 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T10:30:43.176 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T10:30:45.474 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T10:30:45.592 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : kernel-5.14.0-434.el9.x86_64 4/4 2024-03-29T10:30:50.313 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T10:31:12.758 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T10:31:12.758 INFO:teuthology.orchestra.run.smithi196.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-29T10:31:12.759 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-29T10:31:12.801 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 4/4 2024-03-29T10:31:13.166 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: kernel-5.14.0-434.el9.x86_64 4/4 2024-03-29T10:31:13.166 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : kernel-5.14.0-434.el9.x86_64 1/4 2024-03-29T10:31:13.166 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T10:31:13.166 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T10:31:13.544 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T10:31:13.544 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-29T10:31:13.544 INFO:teuthology.orchestra.run.smithi196.stdout:Installed: 2024-03-29T10:31:13.544 INFO:teuthology.orchestra.run.smithi196.stdout: kernel-5.14.0-434.el9.x86_64 2024-03-29T10:31:13.545 INFO:teuthology.orchestra.run.smithi196.stdout: kernel-core-5.14.0-434.el9.x86_64 2024-03-29T10:31:13.545 INFO:teuthology.orchestra.run.smithi196.stdout: kernel-modules-5.14.0-434.el9.x86_64 2024-03-29T10:31:13.545 INFO:teuthology.orchestra.run.smithi196.stdout: kernel-modules-core-5.14.0-434.el9.x86_64 2024-03-29T10:31:13.545 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-29T10:31:13.545 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-03-29T10:31:13.870 DEBUG:teuthology.orchestra.run.smithi196:> rpm -q kernel | sort -rV | head -n 1 2024-03-29T10:31:13.969 INFO:teuthology.orchestra.run.smithi196.stdout:kernel-5.14.0-434.el9.x86_64 2024-03-29T10:31:13.969 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-03-29T10:31:13.969 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-434.el9.x86_64 2024-03-29T10:31:13.969 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-29T10:31:13.969 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-29T10:31:13.970 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-03-29T10:31:13.970 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi196.front.sepia.ceph.com, path=None, version=distro) 2024-03-29T10:31:13.970 DEBUG:teuthology.orchestra.run.smithi196:> rpm -q kernel | sort -rV | head -n 1 2024-03-29T10:31:14.049 INFO:teuthology.orchestra.run.smithi196.stdout:kernel-5.14.0-434.el9.x86_64 2024-03-29T10:31:14.049 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-03-29T10:31:14.049 DEBUG:teuthology.orchestra.run.smithi196:> sudo rpm -qi grub2-tools 2024-03-29T10:31:14.083 INFO:teuthology.orchestra.run.smithi196.stdout:Name : grub2-tools 2024-03-29T10:31:14.083 INFO:teuthology.orchestra.run.smithi196.stdout:Epoch : 1 2024-03-29T10:31:14.083 INFO:teuthology.orchestra.run.smithi196.stdout:Version : 2.06 2024-03-29T10:31:14.083 INFO:teuthology.orchestra.run.smithi196.stdout:Release : 68.el9 2024-03-29T10:31:14.084 INFO:teuthology.orchestra.run.smithi196.stdout:Architecture: x86_64 2024-03-29T10:31:14.084 INFO:teuthology.orchestra.run.smithi196.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-29T10:31:14.084 INFO:teuthology.orchestra.run.smithi196.stdout:Group : Unspecified 2024-03-29T10:31:14.084 INFO:teuthology.orchestra.run.smithi196.stdout:Size : 8264254 2024-03-29T10:31:14.084 INFO:teuthology.orchestra.run.smithi196.stdout:License : GPLv3+ 2024-03-29T10:31:14.084 INFO:teuthology.orchestra.run.smithi196.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-29T10:31:14.084 INFO:teuthology.orchestra.run.smithi196.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-29T10:31:14.084 INFO:teuthology.orchestra.run.smithi196.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-29T10:31:14.084 INFO:teuthology.orchestra.run.smithi196.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-29T10:31:14.084 INFO:teuthology.orchestra.run.smithi196.stdout:Packager : builder@centos.org 2024-03-29T10:31:14.084 INFO:teuthology.orchestra.run.smithi196.stdout:Vendor : CentOS 2024-03-29T10:31:14.084 INFO:teuthology.orchestra.run.smithi196.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-29T10:31:14.084 INFO:teuthology.orchestra.run.smithi196.stdout:Summary : Support tools for GRUB. 2024-03-29T10:31:14.084 INFO:teuthology.orchestra.run.smithi196.stdout:Description : 2024-03-29T10:31:14.084 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-29T10:31:14.085 INFO:teuthology.orchestra.run.smithi196.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-29T10:31:14.085 INFO:teuthology.orchestra.run.smithi196.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-29T10:31:14.085 INFO:teuthology.orchestra.run.smithi196.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-29T10:31:14.085 INFO:teuthology.orchestra.run.smithi196.stdout:hardware devices. 2024-03-29T10:31:14.085 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-29T10:31:14.085 INFO:teuthology.orchestra.run.smithi196.stdout:This subpackage provides tools for support of all platforms. 2024-03-29T10:31:14.086 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-29T10:31:14.086 INFO:teuthology.task.kernel:Updating grub on smithi196 to boot 5.14.0-434.el9.x86_64 2024-03-29T10:31:14.086 DEBUG:teuthology.orchestra.run.smithi196:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-29T10:31:14.662 INFO:teuthology.orchestra.run.smithi196.stderr:Generating grub configuration file ... 2024-03-29T10:31:16.085 INFO:teuthology.orchestra.run.smithi196.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-29T10:31:16.113 INFO:teuthology.orchestra.run.smithi196.stderr:done 2024-03-29T10:31:16.115 DEBUG:teuthology.orchestra.run.smithi196:> mktemp 2024-03-29T10:31:16.131 INFO:teuthology.orchestra.run.smithi196.stdout:/tmp/tmp.N9s5KDA8Sw 2024-03-29T10:31:16.131 DEBUG:teuthology.orchestra.run.smithi196:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.N9s5KDA8Sw 2024-03-29T10:31:16.200 DEBUG:teuthology.orchestra.run.smithi196:> sudo chmod 0666 /tmp/tmp.N9s5KDA8Sw 2024-03-29T10:31:16.337 DEBUG:teuthology.orchestra.remote:smithi196:/tmp/tmp.N9s5KDA8Sw is 6KB 2024-03-29T10:31:16.390 DEBUG:teuthology.orchestra.run.smithi196:> rm -fr /tmp/tmp.N9s5KDA8Sw 2024-03-29T10:31:16.404 DEBUG:teuthology.orchestra.run.smithi196:> sudo /bin/ls /boot/loader/entries || true 2024-03-29T10:31:16.468 INFO:teuthology.orchestra.run.smithi196.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-29T10:31:16.469 INFO:teuthology.orchestra.run.smithi196.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-29T10:31:16.469 INFO:teuthology.orchestra.run.smithi196.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-29T10:31:16.469 INFO:teuthology.orchestra.run.smithi196.stdout:15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64.conf 2024-03-29T10:31:16.470 DEBUG:teuthology.orchestra.run.smithi196:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64 2024-03-29T10:31:16.633 DEBUG:teuthology.orchestra.run.smithi196:> sudo shutdown -r now 2024-03-29T10:31:19.819 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T10:31:19.819 INFO:teuthology.orchestra.run.smithi088.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-29T10:31:19.819 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-03-29T10:31:19.862 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 4/4 2024-03-29T10:31:20.239 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: kernel-5.14.0-434.el9.x86_64 4/4 2024-03-29T10:31:20.239 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : kernel-5.14.0-434.el9.x86_64 1/4 2024-03-29T10:31:20.239 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T10:31:20.239 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T10:31:20.615 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T10:31:20.616 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-03-29T10:31:20.616 INFO:teuthology.orchestra.run.smithi088.stdout:Installed: 2024-03-29T10:31:20.616 INFO:teuthology.orchestra.run.smithi088.stdout: kernel-5.14.0-434.el9.x86_64 2024-03-29T10:31:20.616 INFO:teuthology.orchestra.run.smithi088.stdout: kernel-core-5.14.0-434.el9.x86_64 2024-03-29T10:31:20.616 INFO:teuthology.orchestra.run.smithi088.stdout: kernel-modules-5.14.0-434.el9.x86_64 2024-03-29T10:31:20.616 INFO:teuthology.orchestra.run.smithi088.stdout: kernel-modules-core-5.14.0-434.el9.x86_64 2024-03-29T10:31:20.616 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-03-29T10:31:20.616 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2024-03-29T10:31:20.962 DEBUG:teuthology.orchestra.run.smithi088:> rpm -q kernel | sort -rV | head -n 1 2024-03-29T10:31:21.063 INFO:teuthology.orchestra.run.smithi088.stdout:kernel-5.14.0-434.el9.x86_64 2024-03-29T10:31:21.064 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-03-29T10:31:21.064 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-434.el9.x86_64 2024-03-29T10:31:21.064 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-29T10:31:21.064 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-29T10:31:21.064 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-03-29T10:31:21.064 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi088.front.sepia.ceph.com, path=None, version=distro) 2024-03-29T10:31:21.064 DEBUG:teuthology.orchestra.run.smithi088:> rpm -q kernel | sort -rV | head -n 1 2024-03-29T10:31:21.144 INFO:teuthology.orchestra.run.smithi088.stdout:kernel-5.14.0-434.el9.x86_64 2024-03-29T10:31:21.145 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-03-29T10:31:21.145 DEBUG:teuthology.orchestra.run.smithi088:> sudo rpm -qi grub2-tools 2024-03-29T10:31:21.179 INFO:teuthology.orchestra.run.smithi088.stdout:Name : grub2-tools 2024-03-29T10:31:21.179 INFO:teuthology.orchestra.run.smithi088.stdout:Epoch : 1 2024-03-29T10:31:21.179 INFO:teuthology.orchestra.run.smithi088.stdout:Version : 2.06 2024-03-29T10:31:21.179 INFO:teuthology.orchestra.run.smithi088.stdout:Release : 68.el9 2024-03-29T10:31:21.179 INFO:teuthology.orchestra.run.smithi088.stdout:Architecture: x86_64 2024-03-29T10:31:21.179 INFO:teuthology.orchestra.run.smithi088.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-29T10:31:21.179 INFO:teuthology.orchestra.run.smithi088.stdout:Group : Unspecified 2024-03-29T10:31:21.179 INFO:teuthology.orchestra.run.smithi088.stdout:Size : 8264254 2024-03-29T10:31:21.179 INFO:teuthology.orchestra.run.smithi088.stdout:License : GPLv3+ 2024-03-29T10:31:21.180 INFO:teuthology.orchestra.run.smithi088.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-29T10:31:21.180 INFO:teuthology.orchestra.run.smithi088.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-29T10:31:21.180 INFO:teuthology.orchestra.run.smithi088.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-29T10:31:21.180 INFO:teuthology.orchestra.run.smithi088.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-29T10:31:21.180 INFO:teuthology.orchestra.run.smithi088.stdout:Packager : builder@centos.org 2024-03-29T10:31:21.180 INFO:teuthology.orchestra.run.smithi088.stdout:Vendor : CentOS 2024-03-29T10:31:21.180 INFO:teuthology.orchestra.run.smithi088.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-29T10:31:21.180 INFO:teuthology.orchestra.run.smithi088.stdout:Summary : Support tools for GRUB. 2024-03-29T10:31:21.180 INFO:teuthology.orchestra.run.smithi088.stdout:Description : 2024-03-29T10:31:21.180 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-03-29T10:31:21.180 INFO:teuthology.orchestra.run.smithi088.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-29T10:31:21.180 INFO:teuthology.orchestra.run.smithi088.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-29T10:31:21.180 INFO:teuthology.orchestra.run.smithi088.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-29T10:31:21.181 INFO:teuthology.orchestra.run.smithi088.stdout:hardware devices. 2024-03-29T10:31:21.181 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-03-29T10:31:21.181 INFO:teuthology.orchestra.run.smithi088.stdout:This subpackage provides tools for support of all platforms. 2024-03-29T10:31:21.182 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-29T10:31:21.182 INFO:teuthology.task.kernel:Updating grub on smithi088 to boot 5.14.0-434.el9.x86_64 2024-03-29T10:31:21.182 DEBUG:teuthology.orchestra.run.smithi088:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-29T10:31:21.788 INFO:teuthology.orchestra.run.smithi088.stderr:Generating grub configuration file ... 2024-03-29T10:31:23.225 INFO:teuthology.orchestra.run.smithi088.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-29T10:31:23.264 INFO:teuthology.orchestra.run.smithi088.stderr:done 2024-03-29T10:31:23.272 DEBUG:teuthology.orchestra.run.smithi088:> mktemp 2024-03-29T10:31:23.287 INFO:teuthology.orchestra.run.smithi088.stdout:/tmp/tmp.YwugPviXfE 2024-03-29T10:31:23.287 DEBUG:teuthology.orchestra.run.smithi088:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.YwugPviXfE 2024-03-29T10:31:23.356 DEBUG:teuthology.orchestra.run.smithi088:> sudo chmod 0666 /tmp/tmp.YwugPviXfE 2024-03-29T10:31:23.495 DEBUG:teuthology.orchestra.remote:smithi088:/tmp/tmp.YwugPviXfE is 6KB 2024-03-29T10:31:23.546 DEBUG:teuthology.orchestra.run.smithi088:> rm -fr /tmp/tmp.YwugPviXfE 2024-03-29T10:31:23.563 DEBUG:teuthology.orchestra.run.smithi088:> sudo /bin/ls /boot/loader/entries || true 2024-03-29T10:31:23.627 INFO:teuthology.orchestra.run.smithi088.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-29T10:31:23.627 INFO:teuthology.orchestra.run.smithi088.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-29T10:31:23.627 INFO:teuthology.orchestra.run.smithi088.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-29T10:31:23.627 INFO:teuthology.orchestra.run.smithi088.stdout:15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64.conf 2024-03-29T10:31:23.628 DEBUG:teuthology.orchestra.run.smithi088:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64 2024-03-29T10:31:23.779 DEBUG:teuthology.orchestra.run.smithi088:> sudo shutdown -r now 2024-03-29T10:31:46.658 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-03-29T10:31:46.659 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi196.front.sepia.ceph.com' 2024-03-29T10:31:46.659 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi196.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T10:31:53.789 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-03-29T10:31:53.790 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi088.front.sepia.ceph.com' 2024-03-29T10:31:53.790 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi088.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T10:32:05.154 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.196 2024-03-29T10:32:12.322 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.88 2024-03-29T10:32:14.158 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi196.front.sepia.ceph.com' 2024-03-29T10:32:14.159 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi196.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T10:32:21.330 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi088.front.sepia.ceph.com' 2024-03-29T10:32:21.331 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi088.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T10:32:24.386 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.88 2024-03-29T10:32:32.546 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.196 2024-03-29T10:32:36.390 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi088.front.sepia.ceph.com' 2024-03-29T10:32:36.391 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi088.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T10:32:39.454 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.88 2024-03-29T10:32:44.550 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi196.front.sepia.ceph.com' 2024-03-29T10:32:44.551 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi196.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T10:32:47.614 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.196 2024-03-29T10:32:54.462 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi088.front.sepia.ceph.com' 2024-03-29T10:32:54.462 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi088.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T10:33:02.622 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi196.front.sepia.ceph.com' 2024-03-29T10:33:02.623 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi196.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T10:33:02.902 DEBUG:teuthology.orchestra.run.smithi196:> true 2024-03-29T10:33:03.274 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi196.front.sepia.ceph.com' 2024-03-29T10:33:03.275 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-434.el9.x86_64"... 2024-03-29T10:33:03.275 DEBUG:teuthology.orchestra.run.smithi196:> uname -r 2024-03-29T10:33:03.290 INFO:teuthology.orchestra.run.smithi196.stdout:5.14.0-434.el9.x86_64 2024-03-29T10:33:03.290 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-434.el9.x86_64 vs 5.14.0-434.el9.x86_64 2024-03-29T10:33:03.290 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-29T10:33:03.290 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-29T10:33:04.291 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-03-29T10:33:04.291 DEBUG:teuthology.orchestra.run.smithi196:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-29T10:33:04.654 INFO:teuthology.orchestra.run.smithi196.stdout:ttyS1 2024-03-29T10:33:04.690 DEBUG:teuthology.parallel:result is None 2024-03-29T10:33:26.391 DEBUG:teuthology.orchestra.run.smithi088:> true 2024-03-29T10:33:26.876 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi088.front.sepia.ceph.com' 2024-03-29T10:33:26.876 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-434.el9.x86_64"... 2024-03-29T10:33:26.876 DEBUG:teuthology.orchestra.run.smithi088:> uname -r 2024-03-29T10:33:26.890 INFO:teuthology.orchestra.run.smithi088.stdout:5.14.0-434.el9.x86_64 2024-03-29T10:33:26.890 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-434.el9.x86_64 vs 5.14.0-434.el9.x86_64 2024-03-29T10:33:26.890 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-29T10:33:26.890 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-29T10:33:27.891 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-03-29T10:33:27.891 DEBUG:teuthology.orchestra.run.smithi088:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-29T10:33:27.938 INFO:teuthology.orchestra.run.smithi088.stdout:ttyS1 2024-03-29T10:33:27.970 DEBUG:teuthology.parallel:result is None 2024-03-29T10:33:27.971 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-29T10:33:27.978 INFO:teuthology.task.internal:Creating test directory... 2024-03-29T10:33:27.978 DEBUG:teuthology.orchestra.run.smithi088:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-29T10:33:27.981 DEBUG:teuthology.orchestra.run.smithi196:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-29T10:33:27.999 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-29T10:33:28.023 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-29T10:33:28.056 INFO:teuthology.task.internal:Creating archive directory... 2024-03-29T10:33:28.056 DEBUG:teuthology.orchestra.run.smithi088:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-29T10:33:28.058 DEBUG:teuthology.orchestra.run.smithi196:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-29T10:33:28.099 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-29T10:33:28.107 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-29T10:33:28.108 DEBUG:teuthology.orchestra.run.smithi088:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-29T10:33:28.121 DEBUG:teuthology.orchestra.run.smithi196:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-29T10:33:28.152 INFO:teuthology.orchestra.run.smithi088.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T10:33:28.166 INFO:teuthology.orchestra.run.smithi088.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T10:33:28.179 INFO:teuthology.orchestra.run.smithi196.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T10:33:28.194 INFO:teuthology.orchestra.run.smithi196.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T10:33:28.195 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-29T10:33:28.203 INFO:teuthology.task.internal:Configuring sudo... 2024-03-29T10:33:28.203 DEBUG:teuthology.orchestra.run.smithi088:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-29T10:33:28.210 DEBUG:teuthology.orchestra.run.smithi196:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-29T10:33:28.240 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-29T10:33:28.250 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-29T10:33:28.251 DEBUG:teuthology.orchestra.run.smithi088:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-29T10:33:28.277 DEBUG:teuthology.orchestra.run.smithi196:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-29T10:33:28.296 DEBUG:teuthology.orchestra.run.smithi088:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-29T10:33:28.368 DEBUG:teuthology.orchestra.run.smithi088:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-29T10:33:28.426 DEBUG:teuthology.orchestra.run.smithi088:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-29T10:33:28.505 DEBUG:teuthology.orchestra.run.smithi088:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-29T10:33:28.550 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-03-29T10:33:28.551 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-29T10:33:28.617 DEBUG:teuthology.orchestra.run.smithi196:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-29T10:33:28.651 DEBUG:teuthology.orchestra.run.smithi196:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-29T10:33:28.714 DEBUG:teuthology.orchestra.run.smithi196:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-29T10:33:28.796 DEBUG:teuthology.orchestra.run.smithi196:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-29T10:33:28.844 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2024-03-29T10:33:28.844 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-29T10:33:28.913 DEBUG:teuthology.orchestra.run.smithi088:> sudo service rsyslog restart 2024-03-29T10:33:28.915 DEBUG:teuthology.orchestra.run.smithi196:> sudo service rsyslog restart 2024-03-29T10:33:28.955 INFO:teuthology.orchestra.run.smithi088.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-29T10:33:28.995 INFO:teuthology.orchestra.run.smithi196.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-29T10:33:29.439 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-29T10:33:29.455 INFO:teuthology.task.internal:Starting timer... 2024-03-29T10:33:29.455 INFO:teuthology.run_tasks:Running task pcp... 2024-03-29T10:33:29.500 INFO:teuthology.run_tasks:Running task selinux... 2024-03-29T10:33:29.533 DEBUG:teuthology.orchestra.run.smithi088:> sudo service auditd rotate 2024-03-29T10:33:29.598 INFO:teuthology.orchestra.run.smithi088.stdout:Rotating logs: 2024-03-29T10:33:29.600 DEBUG:teuthology.orchestra.run.smithi196:> sudo service auditd rotate 2024-03-29T10:33:29.672 INFO:teuthology.orchestra.run.smithi196.stdout:Rotating logs: 2024-03-29T10:33:29.674 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-29T10:33:29.674 DEBUG:teuthology.orchestra.run.smithi088:> /usr/sbin/getenforce 2024-03-29T10:33:29.712 INFO:teuthology.orchestra.run.smithi088.stdout:Permissive 2024-03-29T10:33:29.712 DEBUG:teuthology.orchestra.run.smithi196:> /usr/sbin/getenforce 2024-03-29T10:33:29.731 INFO:teuthology.orchestra.run.smithi196.stdout:Permissive 2024-03-29T10:33:29.732 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi088.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi196.front.sepia.ceph.com': 'permissive'} 2024-03-29T10:33:29.732 DEBUG:teuthology.orchestra.run.smithi088:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-29T10:33:29.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T10:33:29.780 DEBUG:teuthology.orchestra.run.smithi196:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-29T10:33:29.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T10:33:29.808 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-29T10:33:29.809 DEBUG:teuthology.orchestra.run.smithi088:> sudo /usr/sbin/setenforce permissive 2024-03-29T10:33:29.856 DEBUG:teuthology.orchestra.run.smithi196:> sudo /usr/sbin/setenforce permissive 2024-03-29T10:33:29.900 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-29T10:33:29.910 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-29T10:33:29.927 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-29T10:33:29.928 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi088.front.sepia.ceph.com,smithi196.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-29T10:37:41.796 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi088.front.sepia.ceph.com'), Remote(name='ubuntu@smithi196.front.sepia.ceph.com')] 2024-03-29T10:37:41.797 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi088.front.sepia.ceph.com' 2024-03-29T10:37:41.798 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi088.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T10:37:41.885 DEBUG:teuthology.orchestra.run.smithi088:> true 2024-03-29T10:37:41.965 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi088.front.sepia.ceph.com' 2024-03-29T10:37:41.965 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi196.front.sepia.ceph.com' 2024-03-29T10:37:41.966 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi196.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T10:37:42.055 DEBUG:teuthology.orchestra.run.smithi196:> true 2024-03-29T10:37:42.093 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi196.front.sepia.ceph.com' 2024-03-29T10:37:42.094 INFO:teuthology.run_tasks:Running task clock... 2024-03-29T10:37:42.104 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-29T10:37:42.104 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-29T10:37:42.105 DEBUG:teuthology.orchestra.run.smithi088:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-29T10:37:42.107 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-29T10:37:42.107 DEBUG:teuthology.orchestra.run.smithi196:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-29T10:37:42.136 INFO:teuthology.orchestra.run.smithi088.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-29T10:37:42.152 INFO:teuthology.orchestra.run.smithi088.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-29T10:37:42.166 INFO:teuthology.orchestra.run.smithi196.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-29T10:37:42.181 INFO:teuthology.orchestra.run.smithi088.stderr:sudo: ntpd: command not found 2024-03-29T10:37:42.183 INFO:teuthology.orchestra.run.smithi196.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-29T10:37:42.195 INFO:teuthology.orchestra.run.smithi088.stdout:506 Cannot talk to daemon 2024-03-29T10:37:42.212 INFO:teuthology.orchestra.run.smithi088.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-29T10:37:42.212 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: ntpd: command not found 2024-03-29T10:37:42.226 INFO:teuthology.orchestra.run.smithi196.stdout:506 Cannot talk to daemon 2024-03-29T10:37:42.228 INFO:teuthology.orchestra.run.smithi088.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-29T10:37:42.242 INFO:teuthology.orchestra.run.smithi196.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-29T10:37:42.259 INFO:teuthology.orchestra.run.smithi196.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-29T10:37:42.282 INFO:teuthology.orchestra.run.smithi088.stderr:bash: line 1: ntpq: command not found 2024-03-29T10:37:42.285 INFO:teuthology.orchestra.run.smithi088.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-29T10:37:42.285 INFO:teuthology.orchestra.run.smithi088.stdout:=============================================================================== 2024-03-29T10:37:42.285 INFO:teuthology.orchestra.run.smithi088.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T10:37:42.285 INFO:teuthology.orchestra.run.smithi088.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T10:37:42.285 INFO:teuthology.orchestra.run.smithi088.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T10:37:42.285 INFO:teuthology.orchestra.run.smithi088.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T10:37:42.312 INFO:teuthology.orchestra.run.smithi196.stderr:bash: line 1: ntpq: command not found 2024-03-29T10:37:42.315 INFO:teuthology.orchestra.run.smithi196.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-29T10:37:42.315 INFO:teuthology.orchestra.run.smithi196.stdout:=============================================================================== 2024-03-29T10:37:42.315 INFO:teuthology.orchestra.run.smithi196.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T10:37:42.315 INFO:teuthology.orchestra.run.smithi196.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T10:37:42.315 INFO:teuthology.orchestra.run.smithi196.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T10:37:42.316 INFO:teuthology.orchestra.run.smithi196.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T10:37:42.316 INFO:teuthology.run_tasks:Running task install... 2024-03-29T10:37:42.325 DEBUG:teuthology.task.install:project ceph 2024-03-29T10:37:42.325 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}} 2024-03-29T10:37:42.325 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'} 2024-03-29T10:37:42.326 INFO:teuthology.task.install:Using flavor: default 2024-03-29T10:37:42.331 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-03-29T10:37:42.331 INFO:teuthology.task.install:extra packages: [] 2024-03-29T10:37:42.332 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'tag': None, 'wait_for_package': False} 2024-03-29T10:37:42.332 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-29T10:37:42.334 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'tag': None, 'wait_for_package': False} 2024-03-29T10:37:42.334 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-29T10:37:42.477 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/ 2024-03-29T10:37:42.477 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1439.g5dcb190d 2024-03-29T10:37:42.483 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/ 2024-03-29T10:37:42.483 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1439.g5dcb190d 2024-03-29T10:37:42.586 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-29T10:37:42.586 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-03-29T10:37:42.586 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-29T10:37:42.614 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-03-29T10:37:42.614 DEBUG:teuthology.orchestra.run.smithi088:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/5dcb190da270440ef141d088eb27a9fdb8afc5f2/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-29T10:37:42.651 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-29T10:37:42.652 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2024-03-29T10:37:42.652 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-29T10:37:42.680 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-03-29T10:37:42.680 DEBUG:teuthology.orchestra.run.smithi196:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/5dcb190da270440ef141d088eb27a9fdb8afc5f2/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-29T10:37:42.682 DEBUG:teuthology.orchestra.run.smithi088:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-29T10:37:42.748 DEBUG:teuthology.orchestra.run.smithi196:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-29T10:37:42.761 DEBUG:teuthology.orchestra.run.smithi088:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-29T10:37:42.828 DEBUG:teuthology.orchestra.run.smithi196:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-29T10:37:42.831 INFO:teuthology.orchestra.run.smithi088.stdout:check_obsoletes = 1 2024-03-29T10:37:42.831 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum clean all 2024-03-29T10:37:42.897 INFO:teuthology.orchestra.run.smithi196.stdout:check_obsoletes = 1 2024-03-29T10:37:42.899 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum clean all 2024-03-29T10:37:43.074 INFO:teuthology.orchestra.run.smithi088.stdout:74 files removed 2024-03-29T10:37:43.101 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install ceph-radosgw 2024-03-29T10:37:43.143 INFO:teuthology.orchestra.run.smithi196.stdout:74 files removed 2024-03-29T10:37:43.174 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install ceph-radosgw 2024-03-29T10:37:43.737 INFO:teuthology.orchestra.run.smithi088.stdout:ceph packages for x86_64 243 kB/s | 82 kB 00:00 2024-03-29T10:37:43.856 INFO:teuthology.orchestra.run.smithi196.stdout:ceph packages for x86_64 214 kB/s | 82 kB 00:00 2024-03-29T10:37:44.179 INFO:teuthology.orchestra.run.smithi088.stdout:ceph noarch packages 26 kB/s | 10 kB 00:00 2024-03-29T10:37:44.273 INFO:teuthology.orchestra.run.smithi196.stdout:ceph noarch packages 27 kB/s | 10 kB 00:00 2024-03-29T10:37:44.596 INFO:teuthology.orchestra.run.smithi088.stdout:ceph source packages 5.0 kB/s | 1.9 kB 00:00 2024-03-29T10:37:44.673 INFO:teuthology.orchestra.run.smithi196.stdout:ceph source packages 5.2 kB/s | 1.9 kB 00:00 2024-03-29T10:37:45.165 INFO:teuthology.orchestra.run.smithi088.stdout:CentOS-9 - AppStream 13 MB/s | 6.7 MB 00:00 2024-03-29T10:37:45.267 INFO:teuthology.orchestra.run.smithi196.stdout:CentOS-9 - AppStream 12 MB/s | 6.7 MB 00:00 2024-03-29T10:37:46.659 INFO:teuthology.orchestra.run.smithi196.stdout:CentOS-9 - BaseOS 7.3 MB/s | 2.0 MB 00:00 2024-03-29T10:37:46.666 INFO:teuthology.orchestra.run.smithi088.stdout:CentOS-9 - BaseOS 5.1 MB/s | 2.0 MB 00:00 2024-03-29T10:37:47.492 INFO:teuthology.orchestra.run.smithi196.stdout:CentOS Stream 9 - BaseOS 16 MB/s | 8.0 MB 00:00 2024-03-29T10:37:50.389 INFO:teuthology.orchestra.run.smithi088.stdout:CentOS Stream 9 - BaseOS 2.4 MB/s | 8.0 MB 00:03 2024-03-29T10:37:51.367 INFO:teuthology.orchestra.run.smithi196.stdout:CentOS Stream 9 - AppStream 6.7 MB/s | 19 MB 00:02 2024-03-29T10:37:52.374 INFO:teuthology.orchestra.run.smithi088.stdout:CentOS Stream 9 - AppStream 21 MB/s | 19 MB 00:00 2024-03-29T10:37:56.035 INFO:teuthology.orchestra.run.smithi196.stdout:CentOS Stream 9 - CRB 5.3 MB/s | 6.1 MB 00:01 2024-03-29T10:37:56.599 INFO:teuthology.orchestra.run.smithi088.stdout:CentOS Stream 9 - CRB 8.3 MB/s | 6.1 MB 00:00 2024-03-29T10:37:57.803 INFO:teuthology.orchestra.run.smithi196.stdout:CentOS Stream 9 - Extras packages 26 kB/s | 16 kB 00:00 2024-03-29T10:37:59.050 INFO:teuthology.orchestra.run.smithi088.stdout:CentOS Stream 9 - Extras packages 15 kB/s | 16 kB 00:01 2024-03-29T10:37:59.237 INFO:teuthology.orchestra.run.smithi196.stdout:Extra Packages for Enterprise Linux 16 MB/s | 21 MB 00:01 2024-03-29T10:38:00.534 INFO:teuthology.orchestra.run.smithi088.stdout:Extra Packages for Enterprise Linux 15 MB/s | 21 MB 00:01 2024-03-29T10:38:04.245 INFO:teuthology.orchestra.run.smithi196.stdout:lab-extras 27 kB/s | 1.7 kB 00:00 2024-03-29T10:38:05.394 INFO:teuthology.orchestra.run.smithi088.stdout:lab-extras 25 kB/s | 1.7 kB 00:00 2024-03-29T10:38:06.887 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-03-29T10:38:06.891 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-29T10:38:06.891 INFO:teuthology.orchestra.run.smithi196.stdout: Package Arch Version Repository Size 2024-03-29T10:38:06.891 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-29T10:38:06.891 INFO:teuthology.orchestra.run.smithi196.stdout:Installing: 2024-03-29T10:38:06.891 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-radosgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 9.7 M 2024-03-29T10:38:06.891 INFO:teuthology.orchestra.run.smithi196.stdout:Upgrading: 2024-03-29T10:38:06.891 INFO:teuthology.orchestra.run.smithi196.stdout: librados2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.4 M 2024-03-29T10:38:06.891 INFO:teuthology.orchestra.run.smithi196.stdout: librbd1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-29T10:38:06.891 INFO:teuthology.orchestra.run.smithi196.stdout:Installing dependencies: 2024-03-29T10:38:06.891 INFO:teuthology.orchestra.run.smithi196.stdout: boost-program-options 2024-03-29T10:38:06.891 INFO:teuthology.orchestra.run.smithi196.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-29T10:38:06.891 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-base x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.3 M 2024-03-29T10:38:06.892 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-29T10:38:06.892 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-selinux x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 25 k 2024-03-29T10:38:06.892 INFO:teuthology.orchestra.run.smithi196.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-29T10:38:06.892 INFO:teuthology.orchestra.run.smithi196.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-29T10:38:06.892 INFO:teuthology.orchestra.run.smithi196.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-29T10:38:06.892 INFO:teuthology.orchestra.run.smithi196.stdout: libcephfs2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 742 k 2024-03-29T10:38:06.892 INFO:teuthology.orchestra.run.smithi196.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-29T10:38:06.892 INFO:teuthology.orchestra.run.smithi196.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-29T10:38:06.892 INFO:teuthology.orchestra.run.smithi196.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-29T10:38:06.892 INFO:teuthology.orchestra.run.smithi196.stdout: libradosstriper1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 498 k 2024-03-29T10:38:06.892 INFO:teuthology.orchestra.run.smithi196.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-29T10:38:06.892 INFO:teuthology.orchestra.run.smithi196.stdout: librgw2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.1 M 2024-03-29T10:38:06.892 INFO:teuthology.orchestra.run.smithi196.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-29T10:38:06.892 INFO:teuthology.orchestra.run.smithi196.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-29T10:38:06.893 INFO:teuthology.orchestra.run.smithi196.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-29T10:38:06.893 INFO:teuthology.orchestra.run.smithi196.stdout: python3-ceph-argparse 2024-03-29T10:38:06.893 INFO:teuthology.orchestra.run.smithi196.stdout: x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 45 k 2024-03-29T10:38:06.893 INFO:teuthology.orchestra.run.smithi196.stdout: python3-ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 131 k 2024-03-29T10:38:06.893 INFO:teuthology.orchestra.run.smithi196.stdout: python3-cephfs x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 162 k 2024-03-29T10:38:06.893 INFO:teuthology.orchestra.run.smithi196.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-29T10:38:06.893 INFO:teuthology.orchestra.run.smithi196.stdout: python3-rados x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 320 k 2024-03-29T10:38:06.893 INFO:teuthology.orchestra.run.smithi196.stdout: python3-rbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 298 k 2024-03-29T10:38:06.893 INFO:teuthology.orchestra.run.smithi196.stdout: python3-rgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 k 2024-03-29T10:38:06.893 INFO:teuthology.orchestra.run.smithi196.stdout: qatlib x86_64 24.02.0-1.el9 CentOS-AppStream 220 k 2024-03-29T10:38:06.893 INFO:teuthology.orchestra.run.smithi196.stdout: qatzip-libs x86_64 1.2.0-1.el9 CentOS-AppStream 46 k 2024-03-29T10:38:06.893 INFO:teuthology.orchestra.run.smithi196.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-29T10:38:06.893 INFO:teuthology.orchestra.run.smithi196.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-29T10:38:06.894 INFO:teuthology.orchestra.run.smithi196.stdout:Installing weak dependencies: 2024-03-29T10:38:06.894 INFO:teuthology.orchestra.run.smithi196.stdout: qatlib-service x86_64 24.02.0-1.el9 CentOS-AppStream 35 k 2024-03-29T10:38:06.894 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-29T10:38:06.894 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction Summary 2024-03-29T10:38:06.894 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-29T10:38:06.894 INFO:teuthology.orchestra.run.smithi196.stdout:Install 30 Packages 2024-03-29T10:38:06.894 INFO:teuthology.orchestra.run.smithi196.stdout:Upgrade 2 Packages 2024-03-29T10:38:06.894 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-29T10:38:06.894 INFO:teuthology.orchestra.run.smithi196.stdout:Total download size: 58 M 2024-03-29T10:38:06.895 INFO:teuthology.orchestra.run.smithi196.stdout:Downloading Packages: 2024-03-29T10:38:08.005 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2024-03-29T10:38:08.009 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-03-29T10:38:08.009 INFO:teuthology.orchestra.run.smithi088.stdout: Package Arch Version Repository Size 2024-03-29T10:38:08.009 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-03-29T10:38:08.009 INFO:teuthology.orchestra.run.smithi088.stdout:Installing: 2024-03-29T10:38:08.009 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-radosgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 9.7 M 2024-03-29T10:38:08.009 INFO:teuthology.orchestra.run.smithi088.stdout:Upgrading: 2024-03-29T10:38:08.009 INFO:teuthology.orchestra.run.smithi088.stdout: librados2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.4 M 2024-03-29T10:38:08.009 INFO:teuthology.orchestra.run.smithi088.stdout: librbd1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-29T10:38:08.009 INFO:teuthology.orchestra.run.smithi088.stdout:Installing dependencies: 2024-03-29T10:38:08.009 INFO:teuthology.orchestra.run.smithi088.stdout: boost-program-options 2024-03-29T10:38:08.010 INFO:teuthology.orchestra.run.smithi088.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-29T10:38:08.010 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-base x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.3 M 2024-03-29T10:38:08.010 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-29T10:38:08.010 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-selinux x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 25 k 2024-03-29T10:38:08.010 INFO:teuthology.orchestra.run.smithi088.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-29T10:38:08.010 INFO:teuthology.orchestra.run.smithi088.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-29T10:38:08.010 INFO:teuthology.orchestra.run.smithi088.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-29T10:38:08.010 INFO:teuthology.orchestra.run.smithi088.stdout: libcephfs2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 742 k 2024-03-29T10:38:08.010 INFO:teuthology.orchestra.run.smithi088.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-29T10:38:08.010 INFO:teuthology.orchestra.run.smithi088.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-29T10:38:08.010 INFO:teuthology.orchestra.run.smithi088.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-29T10:38:08.010 INFO:teuthology.orchestra.run.smithi088.stdout: libradosstriper1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 498 k 2024-03-29T10:38:08.010 INFO:teuthology.orchestra.run.smithi088.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-29T10:38:08.011 INFO:teuthology.orchestra.run.smithi088.stdout: librgw2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.1 M 2024-03-29T10:38:08.011 INFO:teuthology.orchestra.run.smithi088.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-29T10:38:08.011 INFO:teuthology.orchestra.run.smithi088.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-29T10:38:08.011 INFO:teuthology.orchestra.run.smithi088.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-29T10:38:08.011 INFO:teuthology.orchestra.run.smithi088.stdout: python3-ceph-argparse 2024-03-29T10:38:08.011 INFO:teuthology.orchestra.run.smithi088.stdout: x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 45 k 2024-03-29T10:38:08.011 INFO:teuthology.orchestra.run.smithi088.stdout: python3-ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 131 k 2024-03-29T10:38:08.011 INFO:teuthology.orchestra.run.smithi088.stdout: python3-cephfs x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 162 k 2024-03-29T10:38:08.011 INFO:teuthology.orchestra.run.smithi088.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-29T10:38:08.011 INFO:teuthology.orchestra.run.smithi088.stdout: python3-rados x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 320 k 2024-03-29T10:38:08.011 INFO:teuthology.orchestra.run.smithi088.stdout: python3-rbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 298 k 2024-03-29T10:38:08.011 INFO:teuthology.orchestra.run.smithi088.stdout: python3-rgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 k 2024-03-29T10:38:08.011 INFO:teuthology.orchestra.run.smithi088.stdout: qatlib x86_64 24.02.0-1.el9 CentOS-AppStream 220 k 2024-03-29T10:38:08.011 INFO:teuthology.orchestra.run.smithi088.stdout: qatzip-libs x86_64 1.2.0-1.el9 CentOS-AppStream 46 k 2024-03-29T10:38:08.012 INFO:teuthology.orchestra.run.smithi088.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-29T10:38:08.012 INFO:teuthology.orchestra.run.smithi088.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-29T10:38:08.012 INFO:teuthology.orchestra.run.smithi088.stdout:Installing weak dependencies: 2024-03-29T10:38:08.012 INFO:teuthology.orchestra.run.smithi088.stdout: qatlib-service x86_64 24.02.0-1.el9 CentOS-AppStream 35 k 2024-03-29T10:38:08.012 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-03-29T10:38:08.012 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction Summary 2024-03-29T10:38:08.012 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-03-29T10:38:08.012 INFO:teuthology.orchestra.run.smithi088.stdout:Install 30 Packages 2024-03-29T10:38:08.012 INFO:teuthology.orchestra.run.smithi088.stdout:Upgrade 2 Packages 2024-03-29T10:38:08.012 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-03-29T10:38:08.013 INFO:teuthology.orchestra.run.smithi088.stdout:Total download size: 58 M 2024-03-29T10:38:08.013 INFO:teuthology.orchestra.run.smithi088.stdout:Downloading Packages: 2024-03-29T10:38:08.437 INFO:teuthology.orchestra.run.smithi196.stdout:(1/32): ceph-base-19.0.0-1439.g5dcb190d.el9.x86 3.7 MB/s | 5.3 MB 00:01 2024-03-29T10:38:08.537 INFO:teuthology.orchestra.run.smithi196.stdout:(2/32): ceph-selinux-19.0.0-1439.g5dcb190d.el9. 252 kB/s | 25 kB 00:00 2024-03-29T10:38:08.746 INFO:teuthology.orchestra.run.smithi196.stdout:(3/32): libcephfs2-19.0.0-1439.g5dcb190d.el9.x8 3.5 MB/s | 742 kB 00:00 2024-03-29T10:38:08.855 INFO:teuthology.orchestra.run.smithi196.stdout:(4/32): libradosstriper1-19.0.0-1439.g5dcb190d. 4.5 MB/s | 498 kB 00:00 2024-03-29T10:38:09.163 INFO:teuthology.orchestra.run.smithi196.stdout:(5/32): ceph-radosgw-19.0.0-1439.g5dcb190d.el9. 4.4 MB/s | 9.7 MB 00:02 2024-03-29T10:38:09.263 INFO:teuthology.orchestra.run.smithi196.stdout:(6/32): python3-ceph-argparse-19.0.0-1439.g5dcb 452 kB/s | 45 kB 00:00 2024-03-29T10:38:09.372 INFO:teuthology.orchestra.run.smithi196.stdout:(7/32): python3-ceph-common-19.0.0-1439.g5dcb19 1.2 MB/s | 131 kB 00:00 2024-03-29T10:38:09.639 INFO:teuthology.orchestra.run.smithi196.stdout:(8/32): ceph-common-19.0.0-1439.g5dcb190d.el9.x 7.5 MB/s | 20 MB 00:02 2024-03-29T10:38:09.664 INFO:teuthology.orchestra.run.smithi196.stdout:(9/32): python3-cephfs-19.0.0-1439.g5dcb190d.el 554 kB/s | 162 kB 00:00 2024-03-29T10:38:09.740 INFO:teuthology.orchestra.run.smithi196.stdout:(10/32): librgw2-19.0.0-1439.g5dcb190d.el9.x86_ 5.8 MB/s | 5.1 MB 00:00 2024-03-29T10:38:09.765 INFO:teuthology.orchestra.run.smithi196.stdout:(11/32): python3-rados-19.0.0-1439.g5dcb190d.el 2.5 MB/s | 320 kB 00:00 2024-03-29T10:38:09.790 INFO:teuthology.orchestra.run.smithi196.stdout:(12/32): python3-rbd-19.0.0-1439.g5dcb190d.el9. 2.3 MB/s | 298 kB 00:00 2024-03-29T10:38:09.824 INFO:teuthology.orchestra.run.smithi196.stdout:(13/32): python3-rgw-19.0.0-1439.g5dcb190d.el9. 1.2 MB/s | 99 kB 00:00 2024-03-29T10:38:09.957 INFO:teuthology.orchestra.run.smithi196.stdout:(14/32): boost-program-options-1.75.0-8.el9.x86 550 kB/s | 106 kB 00:00 2024-03-29T10:38:09.983 INFO:teuthology.orchestra.run.smithi196.stdout:(15/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 826 kB/s | 159 kB 00:00 2024-03-29T10:38:10.007 INFO:teuthology.orchestra.run.smithi196.stdout:(16/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 325 kB/s | 44 kB 00:00 2024-03-29T10:38:10.141 INFO:teuthology.orchestra.run.smithi196.stdout:(17/32): librdkafka-1.6.1-102.el9.x86_64.rpm 3.5 MB/s | 661 kB 00:00 2024-03-29T10:38:10.166 INFO:teuthology.orchestra.run.smithi196.stdout:(18/32): python3-prettytable-0.7.2-27.el9.noarc 260 kB/s | 41 kB 00:00 2024-03-29T10:38:10.191 INFO:teuthology.orchestra.run.smithi196.stdout:(19/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 1.5 MB/s | 282 kB 00:00 2024-03-29T10:38:10.217 INFO:teuthology.orchestra.run.smithi196.stdout:(20/32): qatlib-24.02.0-1.el9.x86_64.rpm 2.8 MB/s | 220 kB 00:00 2024-03-29T10:38:10.242 INFO:teuthology.orchestra.run.smithi196.stdout:(21/32): qatlib-service-24.02.0-1.el9.x86_64.rp 466 kB/s | 35 kB 00:00 2024-03-29T10:38:10.267 INFO:teuthology.orchestra.run.smithi196.stdout:(22/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 922 kB/s | 46 kB 00:00 2024-03-29T10:38:10.559 INFO:teuthology.orchestra.run.smithi196.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 89 kB/s | 26 kB 00:00 2024-03-29T10:38:10.676 INFO:teuthology.orchestra.run.smithi196.stdout:(24/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 673 kB/s | 309 kB 00:00 2024-03-29T10:38:10.702 INFO:teuthology.orchestra.run.smithi196.stdout:(25/32): liboath-2.6.7-2.el9.x86_64.rpm 342 kB/s | 49 kB 00:00 2024-03-29T10:38:10.752 INFO:teuthology.orchestra.run.smithi196.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 895 kB/s | 67 kB 00:00 2024-03-29T10:38:10.834 INFO:teuthology.orchestra.run.smithi088.stdout:(1/32): ceph-base-19.0.0-1439.g5dcb190d.el9.x86 2.0 MB/s | 5.3 MB 00:02 2024-03-29T10:38:10.852 INFO:teuthology.orchestra.run.smithi196.stdout:(27/32): re2-20211101-3.el9.x86_64.rpm 1.9 MB/s | 192 kB 00:00 2024-03-29T10:38:10.901 INFO:teuthology.orchestra.run.smithi088.stdout:(2/32): ceph-selinux-19.0.0-1439.g5dcb190d.el9. 379 kB/s | 25 kB 00:00 2024-03-29T10:38:10.977 INFO:teuthology.orchestra.run.smithi196.stdout:(28/32): libarrow-9.0.0-10.el9.x86_64.rpm 6.0 MB/s | 4.4 MB 00:00 2024-03-29T10:38:11.011 INFO:teuthology.orchestra.run.smithi196.stdout:(29/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 2.6 MB/s | 838 kB 00:00 2024-03-29T10:38:11.111 INFO:teuthology.orchestra.run.smithi196.stdout:(30/32): thrift-0.15.0-2.el9.x86_64.rpm 6.1 MB/s | 1.6 MB 00:00 2024-03-29T10:38:11.160 INFO:teuthology.orchestra.run.smithi088.stdout:(3/32): libcephfs2-19.0.0-1439.g5dcb190d.el9.x8 2.8 MB/s | 742 kB 00:00 2024-03-29T10:38:11.369 INFO:teuthology.orchestra.run.smithi088.stdout:(4/32): ceph-common-19.0.0-1439.g5dcb190d.el9.x 6.2 MB/s | 20 MB 00:03 2024-03-29T10:38:11.432 INFO:teuthology.orchestra.run.smithi088.stdout:(5/32): libradosstriper1-19.0.0-1439.g5dcb190d. 1.8 MB/s | 498 kB 00:00 2024-03-29T10:38:11.870 INFO:teuthology.orchestra.run.smithi196.stdout:(31/32): librbd1-19.0.0-1439.g5dcb190d.el9.x86_ 3.6 MB/s | 3.1 MB 00:00 2024-03-29T10:38:11.902 INFO:teuthology.orchestra.run.smithi088.stdout:(6/32): librgw2-19.0.0-1439.g5dcb190d.el9.x86_6 9.6 MB/s | 5.1 MB 00:00 2024-03-29T10:38:11.920 INFO:teuthology.orchestra.run.smithi088.stdout:(7/32): python3-ceph-argparse-19.0.0-1439.g5dcb 92 kB/s | 45 kB 00:00 2024-03-29T10:38:11.962 INFO:teuthology.orchestra.run.smithi088.stdout:(8/32): python3-ceph-common-19.0.0-1439.g5dcb19 2.2 MB/s | 131 kB 00:00 2024-03-29T10:38:11.987 INFO:teuthology.orchestra.run.smithi088.stdout:(9/32): python3-cephfs-19.0.0-1439.g5dcb190d.el 2.4 MB/s | 162 kB 00:00 2024-03-29T10:38:12.010 INFO:teuthology.orchestra.run.smithi196.stdout:(32/32): librados2-19.0.0-1439.g5dcb190d.el9.x8 3.3 MB/s | 3.4 MB 00:01 2024-03-29T10:38:12.016 INFO:teuthology.orchestra.run.smithi196.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:38:12.016 INFO:teuthology.orchestra.run.smithi196.stdout:Total 11 MB/s | 58 MB 00:05 2024-03-29T10:38:12.017 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction check 2024-03-29T10:38:12.037 INFO:teuthology.orchestra.run.smithi088.stdout:(10/32): python3-rbd-19.0.0-1439.g5dcb190d.el9. 5.8 MB/s | 298 kB 00:00 2024-03-29T10:38:12.051 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction check succeeded. 2024-03-29T10:38:12.051 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction test 2024-03-29T10:38:12.062 INFO:teuthology.orchestra.run.smithi088.stdout:(11/32): python3-rados-19.0.0-1439.g5dcb190d.el 3.1 MB/s | 320 kB 00:00 2024-03-29T10:38:12.088 INFO:teuthology.orchestra.run.smithi088.stdout:(12/32): python3-rgw-19.0.0-1439.g5dcb190d.el9. 1.9 MB/s | 99 kB 00:00 2024-03-29T10:38:12.204 INFO:teuthology.orchestra.run.smithi088.stdout:(13/32): boost-program-options-1.75.0-8.el9.x86 747 kB/s | 106 kB 00:00 2024-03-29T10:38:12.230 INFO:teuthology.orchestra.run.smithi088.stdout:(14/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.3 MB/s | 159 kB 00:00 2024-03-29T10:38:12.247 INFO:teuthology.orchestra.run.smithi088.stdout:(15/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.0 MB/s | 44 kB 00:00 2024-03-29T10:38:12.297 INFO:teuthology.orchestra.run.smithi088.stdout:(16/32): librdkafka-1.6.1-102.el9.x86_64.rpm 9.7 MB/s | 661 kB 00:00 2024-03-29T10:38:12.322 INFO:teuthology.orchestra.run.smithi088.stdout:(17/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.7 MB/s | 282 kB 00:00 2024-03-29T10:38:12.339 INFO:teuthology.orchestra.run.smithi088.stdout:(18/32): python3-prettytable-0.7.2-27.el9.noarc 993 kB/s | 41 kB 00:00 2024-03-29T10:38:12.364 INFO:teuthology.orchestra.run.smithi088.stdout:(19/32): qatlib-24.02.0-1.el9.x86_64.rpm 5.1 MB/s | 220 kB 00:00 2024-03-29T10:38:12.381 INFO:teuthology.orchestra.run.smithi088.stdout:(20/32): qatlib-service-24.02.0-1.el9.x86_64.rp 841 kB/s | 35 kB 00:00 2024-03-29T10:38:12.399 INFO:teuthology.orchestra.run.smithi088.stdout:(21/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 1.3 MB/s | 46 kB 00:00 2024-03-29T10:38:12.516 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction test succeeded. 2024-03-29T10:38:12.516 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction 2024-03-29T10:38:12.915 INFO:teuthology.orchestra.run.smithi088.stdout:(22/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 579 kB/s | 309 kB 00:00 2024-03-29T10:38:13.208 INFO:teuthology.orchestra.run.smithi088.stdout:(23/32): ceph-radosgw-19.0.0-1439.g5dcb190d.el9 1.9 MB/s | 9.7 MB 00:05 2024-03-29T10:38:13.224 INFO:teuthology.orchestra.run.smithi088.stdout:(24/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 84 kB/s | 26 kB 00:00 2024-03-29T10:38:13.558 INFO:teuthology.orchestra.run.smithi088.stdout:(25/32): liboath-2.6.7-2.el9.x86_64.rpm 139 kB/s | 49 kB 00:00 2024-03-29T10:38:13.926 INFO:teuthology.orchestra.run.smithi088.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 188 kB/s | 67 kB 00:00 2024-03-29T10:38:13.926 INFO:teuthology.orchestra.run.smithi088.stdout:(27/32): libarrow-9.0.0-10.el9.x86_64.rpm 3.5 MB/s | 4.4 MB 00:01 2024-03-29T10:38:13.957 INFO:teuthology.orchestra.run.smithi196.stdout: Preparing : 1/1 2024-03-29T10:38:14.017 INFO:teuthology.orchestra.run.smithi088.stdout:(28/32): re2-20211101-3.el9.x86_64.rpm 442 kB/s | 192 kB 00:00 2024-03-29T10:38:14.043 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-29T10:38:14.074 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-29T10:38:14.100 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-29T10:38:14.122 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-29T10:38:14.139 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-29T10:38:14.167 INFO:teuthology.orchestra.run.smithi088.stdout:(29/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 1.3 MB/s | 838 kB 00:00 2024-03-29T10:38:14.322 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-29T10:38:14.377 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-29T10:38:14.443 INFO:teuthology.orchestra.run.smithi088.stdout:(30/32): thrift-0.15.0-2.el9.x86_64.rpm 2.0 MB/s | 1.6 MB 00:00 2024-03-29T10:38:14.501 INFO:teuthology.orchestra.run.smithi088.stdout:(31/32): librbd1-19.0.0-1439.g5dcb190d.el9.x86_ 9.4 MB/s | 3.1 MB 00:00 2024-03-29T10:38:14.710 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-29T10:38:14.743 INFO:teuthology.orchestra.run.smithi088.stdout:(32/32): librados2-19.0.0-1439.g5dcb190d.el9.x8 4.7 MB/s | 3.4 MB 00:00 2024-03-29T10:38:14.748 INFO:teuthology.orchestra.run.smithi088.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:38:14.748 INFO:teuthology.orchestra.run.smithi088.stdout:Total 8.6 MB/s | 58 MB 00:06 2024-03-29T10:38:14.748 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction check 2024-03-29T10:38:14.784 INFO:teuthology.orchestra.run.smithi196.stdout: Upgrading : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T10:38:14.786 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction check succeeded. 2024-03-29T10:38:14.786 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction test 2024-03-29T10:38:14.875 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T10:38:14.972 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 8/34 2024-03-29T10:38:14.994 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-29T10:38:15.040 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-29T10:38:15.069 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-29T10:38:15.112 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 11/34 2024-03-29T10:38:15.181 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-29T10:38:15.205 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-29T10:38:15.251 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-29T10:38:15.265 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction test succeeded. 2024-03-29T10:38:15.266 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction 2024-03-29T10:38:15.278 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-03-29T10:38:15.386 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-29T10:38:15.421 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-29T10:38:15.461 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-29T10:38:15.494 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-29T10:38:15.523 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-29T10:38:15.555 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-29T10:38:15.884 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-29T10:38:15.910 INFO:teuthology.orchestra.run.smithi196.stdout: Upgrading : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-29T10:38:16.001 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-29T10:38:16.032 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 23/34 2024-03-29T10:38:16.110 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-29T10:38:16.607 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-29T10:38:16.629 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-29T10:38:16.692 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-29T10:38:16.710 INFO:teuthology.orchestra.run.smithi088.stdout: Preparing : 1/1 2024-03-29T10:38:16.761 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 27/34 2024-03-29T10:38:16.790 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 28/34 2024-03-29T10:38:16.803 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-29T10:38:16.831 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-29T10:38:16.858 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-29T10:38:16.887 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-29T10:38:16.905 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-29T10:38:17.062 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-29T10:38:17.118 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-29T10:38:17.488 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-29T10:38:17.524 INFO:teuthology.orchestra.run.smithi088.stdout: Upgrading : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T10:38:17.617 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T10:38:17.707 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 8/34 2024-03-29T10:38:17.734 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-29T10:38:17.797 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-29T10:38:17.834 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-29T10:38:17.878 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 11/34 2024-03-29T10:38:17.946 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-29T10:38:17.970 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-29T10:38:18.034 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-29T10:38:18.069 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-03-29T10:38:18.178 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-29T10:38:18.228 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-29T10:38:18.269 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-29T10:38:18.318 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-29T10:38:18.347 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-29T10:38:18.379 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-29T10:38:18.694 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-29T10:38:18.726 INFO:teuthology.orchestra.run.smithi088.stdout: Upgrading : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-29T10:38:18.816 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-29T10:38:18.856 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 23/34 2024-03-29T10:38:18.895 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-29T10:38:18.900 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T10:38:18.944 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T10:38:19.350 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-29T10:38:19.370 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-29T10:38:19.442 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-29T10:38:19.505 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T10:38:19.509 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 27/34 2024-03-29T10:38:19.548 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 28/34 2024-03-29T10:38:19.548 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-29T10:38:19.636 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-29T10:38:19.636 INFO:teuthology.orchestra.run.smithi196.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-29T10:38:19.637 INFO:teuthology.orchestra.run.smithi196.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-29T10:38:19.637 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-29T10:38:19.674 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T10:38:21.768 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T10:38:21.819 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T10:38:22.373 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T10:38:22.414 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-29T10:38:22.489 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-29T10:38:22.490 INFO:teuthology.orchestra.run.smithi088.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-29T10:38:22.490 INFO:teuthology.orchestra.run.smithi088.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-29T10:38:22.490 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-03-29T10:38:22.532 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T10:38:29.174 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T10:38:29.174 INFO:teuthology.orchestra.run.smithi196.stdout:skipping the directory /sys 2024-03-29T10:38:29.174 INFO:teuthology.orchestra.run.smithi196.stdout:skipping the directory /proc 2024-03-29T10:38:29.174 INFO:teuthology.orchestra.run.smithi196.stdout:skipping the directory /mnt 2024-03-29T10:38:29.175 INFO:teuthology.orchestra.run.smithi196.stdout:skipping the directory /var/tmp 2024-03-29T10:38:29.175 INFO:teuthology.orchestra.run.smithi196.stdout:skipping the directory /home 2024-03-29T10:38:29.175 INFO:teuthology.orchestra.run.smithi196.stdout:skipping the directory /root 2024-03-29T10:38:29.175 INFO:teuthology.orchestra.run.smithi196.stdout:skipping the directory /tmp 2024-03-29T10:38:29.175 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-29T10:38:30.076 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-29T10:38:30.101 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-29T10:38:30.101 INFO:teuthology.orchestra.run.smithi196.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:38:30.101 INFO:teuthology.orchestra.run.smithi196.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-29T10:38:30.102 INFO:teuthology.orchestra.run.smithi196.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-29T10:38:30.102 INFO:teuthology.orchestra.run.smithi196.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-29T10:38:30.102 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-29T10:38:30.103 INFO:teuthology.orchestra.run.smithi196.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-29T10:38:30.188 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-29T10:38:30.188 INFO:teuthology.orchestra.run.smithi196.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T10:38:31.323 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T10:38:31.324 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/34 2024-03-29T10:38:31.324 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/34 2024-03-29T10:38:31.324 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/34 2024-03-29T10:38:31.324 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/34 2024-03-29T10:38:31.324 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/34 2024-03-29T10:38:31.324 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 6/34 2024-03-29T10:38:31.324 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T10:38:31.324 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 8/34 2024-03-29T10:38:31.324 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 9/34 2024-03-29T10:38:31.324 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 10/34 2024-03-29T10:38:31.324 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 11/34 2024-03-29T10:38:31.325 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-29T10:38:31.325 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 13/34 2024-03-29T10:38:31.325 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-29T10:38:31.325 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-29T10:38:31.325 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-29T10:38:31.325 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-29T10:38:31.325 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-29T10:38:31.325 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-29T10:38:31.325 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-03-29T10:38:31.325 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-03-29T10:38:31.325 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-03-29T10:38:31.325 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-29T10:38:31.325 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-29T10:38:31.326 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-29T10:38:31.326 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-29T10:38:31.326 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-29T10:38:31.326 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-29T10:38:31.326 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-29T10:38:31.326 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-29T10:38:31.327 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T10:38:31.327 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-29T10:38:31.327 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/34 2024-03-29T10:38:31.937 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T10:38:31.937 INFO:teuthology.orchestra.run.smithi088.stdout:skipping the directory /sys 2024-03-29T10:38:31.937 INFO:teuthology.orchestra.run.smithi088.stdout:skipping the directory /proc 2024-03-29T10:38:31.937 INFO:teuthology.orchestra.run.smithi088.stdout:skipping the directory /mnt 2024-03-29T10:38:31.937 INFO:teuthology.orchestra.run.smithi088.stdout:skipping the directory /var/tmp 2024-03-29T10:38:31.937 INFO:teuthology.orchestra.run.smithi088.stdout:skipping the directory /home 2024-03-29T10:38:31.938 INFO:teuthology.orchestra.run.smithi088.stdout:skipping the directory /root 2024-03-29T10:38:31.938 INFO:teuthology.orchestra.run.smithi088.stdout:skipping the directory /tmp 2024-03-29T10:38:31.938 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-03-29T10:38:32.258 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T10:38:32.258 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-29T10:38:32.258 INFO:teuthology.orchestra.run.smithi196.stdout:Upgraded: 2024-03-29T10:38:32.258 INFO:teuthology.orchestra.run.smithi196.stdout: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:38:32.258 INFO:teuthology.orchestra.run.smithi196.stdout: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:38:32.258 INFO:teuthology.orchestra.run.smithi196.stdout:Installed: 2024-03-29T10:38:32.258 INFO:teuthology.orchestra.run.smithi196.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-29T10:38:32.258 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:38:32.259 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:38:32.259 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:38:32.259 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:38:32.259 INFO:teuthology.orchestra.run.smithi196.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-29T10:38:32.259 INFO:teuthology.orchestra.run.smithi196.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-29T10:38:32.259 INFO:teuthology.orchestra.run.smithi196.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-29T10:38:32.259 INFO:teuthology.orchestra.run.smithi196.stdout: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:38:32.259 INFO:teuthology.orchestra.run.smithi196.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-29T10:38:32.259 INFO:teuthology.orchestra.run.smithi196.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-29T10:38:32.259 INFO:teuthology.orchestra.run.smithi196.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-29T10:38:32.259 INFO:teuthology.orchestra.run.smithi196.stdout: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:38:32.259 INFO:teuthology.orchestra.run.smithi196.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-29T10:38:32.259 INFO:teuthology.orchestra.run.smithi196.stdout: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:38:32.259 INFO:teuthology.orchestra.run.smithi196.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-29T10:38:32.259 INFO:teuthology.orchestra.run.smithi196.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-29T10:38:32.260 INFO:teuthology.orchestra.run.smithi196.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-29T10:38:32.260 INFO:teuthology.orchestra.run.smithi196.stdout: python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:38:32.260 INFO:teuthology.orchestra.run.smithi196.stdout: python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:38:32.260 INFO:teuthology.orchestra.run.smithi196.stdout: python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:38:32.260 INFO:teuthology.orchestra.run.smithi196.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-29T10:38:32.260 INFO:teuthology.orchestra.run.smithi196.stdout: python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:38:32.260 INFO:teuthology.orchestra.run.smithi196.stdout: python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:38:32.260 INFO:teuthology.orchestra.run.smithi196.stdout: python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:38:32.260 INFO:teuthology.orchestra.run.smithi196.stdout: qatlib-24.02.0-1.el9.x86_64 2024-03-29T10:38:32.260 INFO:teuthology.orchestra.run.smithi196.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-03-29T10:38:32.260 INFO:teuthology.orchestra.run.smithi196.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-03-29T10:38:32.260 INFO:teuthology.orchestra.run.smithi196.stdout: re2-1:20211101-3.el9.x86_64 2024-03-29T10:38:32.260 INFO:teuthology.orchestra.run.smithi196.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-29T10:38:32.260 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-29T10:38:32.261 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-03-29T10:38:32.498 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install ceph-test 2024-03-29T10:38:32.860 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-29T10:38:32.884 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-29T10:38:32.884 INFO:teuthology.orchestra.run.smithi088.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:38:32.884 INFO:teuthology.orchestra.run.smithi088.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-29T10:38:32.885 INFO:teuthology.orchestra.run.smithi088.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-29T10:38:32.885 INFO:teuthology.orchestra.run.smithi088.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-29T10:38:32.885 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-03-29T10:38:32.886 INFO:teuthology.orchestra.run.smithi088.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-29T10:38:32.963 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-29T10:38:32.963 INFO:teuthology.orchestra.run.smithi088.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T10:38:33.055 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:00:29 ago on Fri 29 Mar 2024 10:38:04 AM UTC. 2024-03-29T10:38:33.206 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-03-29T10:38:33.207 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-29T10:38:33.207 INFO:teuthology.orchestra.run.smithi196.stdout: Package Arch Version Repository Size 2024-03-29T10:38:33.207 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-29T10:38:33.207 INFO:teuthology.orchestra.run.smithi196.stdout:Installing: 2024-03-29T10:38:33.207 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-test x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 47 M 2024-03-29T10:38:33.207 INFO:teuthology.orchestra.run.smithi196.stdout:Installing dependencies: 2024-03-29T10:38:33.207 INFO:teuthology.orchestra.run.smithi196.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-29T10:38:33.207 INFO:teuthology.orchestra.run.smithi196.stdout: libcephsqlite x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 163 k 2024-03-29T10:38:33.207 INFO:teuthology.orchestra.run.smithi196.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-29T10:38:33.207 INFO:teuthology.orchestra.run.smithi196.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-29T10:38:33.207 INFO:teuthology.orchestra.run.smithi196.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-29T10:38:33.208 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-29T10:38:33.208 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction Summary 2024-03-29T10:38:33.208 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-29T10:38:33.208 INFO:teuthology.orchestra.run.smithi196.stdout:Install 6 Packages 2024-03-29T10:38:33.208 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-29T10:38:33.208 INFO:teuthology.orchestra.run.smithi196.stdout:Total download size: 48 M 2024-03-29T10:38:33.209 INFO:teuthology.orchestra.run.smithi196.stdout:Installed size: 204 M 2024-03-29T10:38:33.209 INFO:teuthology.orchestra.run.smithi196.stdout:Downloading Packages: 2024-03-29T10:38:33.460 INFO:teuthology.orchestra.run.smithi196.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.1 MB/s | 217 kB 00:00 2024-03-29T10:38:33.510 INFO:teuthology.orchestra.run.smithi196.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 5.9 MB/s | 300 kB 00:00 2024-03-29T10:38:33.535 INFO:teuthology.orchestra.run.smithi196.stdout:(3/6): libcephsqlite-19.0.0-1439.g5dcb190d.el9. 605 kB/s | 163 kB 00:00 2024-03-29T10:38:33.561 INFO:teuthology.orchestra.run.smithi196.stdout:(4/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.2 MB/s | 63 kB 00:00 2024-03-29T10:38:33.703 INFO:teuthology.orchestra.run.smithi196.stdout:(5/6): jq-1.6-16.el9.x86_64.rpm 1.1 MB/s | 187 kB 00:00 2024-03-29T10:38:34.109 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T10:38:34.109 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/34 2024-03-29T10:38:34.109 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/34 2024-03-29T10:38:34.109 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/34 2024-03-29T10:38:34.109 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/34 2024-03-29T10:38:34.110 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/34 2024-03-29T10:38:34.110 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 6/34 2024-03-29T10:38:34.110 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T10:38:34.110 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 8/34 2024-03-29T10:38:34.110 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 9/34 2024-03-29T10:38:34.111 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 10/34 2024-03-29T10:38:34.111 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 11/34 2024-03-29T10:38:34.111 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-29T10:38:34.111 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 13/34 2024-03-29T10:38:34.111 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-29T10:38:34.111 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-29T10:38:34.111 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-29T10:38:34.112 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-29T10:38:34.112 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-29T10:38:34.112 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-29T10:38:34.112 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-03-29T10:38:34.112 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-03-29T10:38:34.112 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-03-29T10:38:34.112 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-29T10:38:34.113 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-29T10:38:34.113 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-29T10:38:34.113 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-29T10:38:34.113 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-29T10:38:34.113 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-29T10:38:34.113 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-29T10:38:34.113 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-29T10:38:34.114 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T10:38:34.114 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-29T10:38:34.114 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/34 2024-03-29T10:38:35.058 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T10:38:35.059 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-03-29T10:38:35.059 INFO:teuthology.orchestra.run.smithi088.stdout:Upgraded: 2024-03-29T10:38:35.059 INFO:teuthology.orchestra.run.smithi088.stdout: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:38:35.059 INFO:teuthology.orchestra.run.smithi088.stdout: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:38:35.059 INFO:teuthology.orchestra.run.smithi088.stdout:Installed: 2024-03-29T10:38:35.059 INFO:teuthology.orchestra.run.smithi088.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-29T10:38:35.059 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:38:35.059 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:38:35.059 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:38:35.059 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:38:35.059 INFO:teuthology.orchestra.run.smithi088.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-29T10:38:35.059 INFO:teuthology.orchestra.run.smithi088.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-29T10:38:35.060 INFO:teuthology.orchestra.run.smithi088.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-29T10:38:35.060 INFO:teuthology.orchestra.run.smithi088.stdout: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:38:35.060 INFO:teuthology.orchestra.run.smithi088.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-29T10:38:35.060 INFO:teuthology.orchestra.run.smithi088.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-29T10:38:35.060 INFO:teuthology.orchestra.run.smithi088.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-29T10:38:35.060 INFO:teuthology.orchestra.run.smithi088.stdout: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:38:35.060 INFO:teuthology.orchestra.run.smithi088.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-29T10:38:35.060 INFO:teuthology.orchestra.run.smithi088.stdout: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:38:35.060 INFO:teuthology.orchestra.run.smithi088.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-29T10:38:35.060 INFO:teuthology.orchestra.run.smithi088.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-29T10:38:35.060 INFO:teuthology.orchestra.run.smithi088.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-29T10:38:35.060 INFO:teuthology.orchestra.run.smithi088.stdout: python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:38:35.060 INFO:teuthology.orchestra.run.smithi088.stdout: python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:38:35.060 INFO:teuthology.orchestra.run.smithi088.stdout: python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:38:35.061 INFO:teuthology.orchestra.run.smithi088.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-29T10:38:35.061 INFO:teuthology.orchestra.run.smithi088.stdout: python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:38:35.061 INFO:teuthology.orchestra.run.smithi088.stdout: python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:38:35.061 INFO:teuthology.orchestra.run.smithi088.stdout: python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:38:35.061 INFO:teuthology.orchestra.run.smithi088.stdout: qatlib-24.02.0-1.el9.x86_64 2024-03-29T10:38:35.061 INFO:teuthology.orchestra.run.smithi088.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-03-29T10:38:35.061 INFO:teuthology.orchestra.run.smithi088.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-03-29T10:38:35.061 INFO:teuthology.orchestra.run.smithi088.stdout: re2-1:20211101-3.el9.x86_64 2024-03-29T10:38:35.061 INFO:teuthology.orchestra.run.smithi088.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-29T10:38:35.061 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-03-29T10:38:35.061 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2024-03-29T10:38:35.431 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install ceph-test 2024-03-29T10:38:35.845 INFO:teuthology.orchestra.run.smithi196.stdout:(6/6): ceph-test-19.0.0-1439.g5dcb190d.el9.x86_ 18 MB/s | 47 MB 00:02 2024-03-29T10:38:35.848 INFO:teuthology.orchestra.run.smithi196.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:38:35.848 INFO:teuthology.orchestra.run.smithi196.stdout:Total 18 MB/s | 48 MB 00:02 2024-03-29T10:38:35.862 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction check 2024-03-29T10:38:35.884 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction check succeeded. 2024-03-29T10:38:35.884 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction test 2024-03-29T10:38:35.980 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:00:30 ago on Fri 29 Mar 2024 10:38:05 AM UTC. 2024-03-29T10:38:36.129 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2024-03-29T10:38:36.130 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-03-29T10:38:36.130 INFO:teuthology.orchestra.run.smithi088.stdout: Package Arch Version Repository Size 2024-03-29T10:38:36.130 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-03-29T10:38:36.130 INFO:teuthology.orchestra.run.smithi088.stdout:Installing: 2024-03-29T10:38:36.130 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-test x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 47 M 2024-03-29T10:38:36.131 INFO:teuthology.orchestra.run.smithi088.stdout:Installing dependencies: 2024-03-29T10:38:36.131 INFO:teuthology.orchestra.run.smithi088.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-29T10:38:36.131 INFO:teuthology.orchestra.run.smithi088.stdout: libcephsqlite x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 163 k 2024-03-29T10:38:36.131 INFO:teuthology.orchestra.run.smithi088.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-29T10:38:36.131 INFO:teuthology.orchestra.run.smithi088.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-29T10:38:36.131 INFO:teuthology.orchestra.run.smithi088.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-29T10:38:36.131 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-03-29T10:38:36.131 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction Summary 2024-03-29T10:38:36.131 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-03-29T10:38:36.131 INFO:teuthology.orchestra.run.smithi088.stdout:Install 6 Packages 2024-03-29T10:38:36.131 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-03-29T10:38:36.132 INFO:teuthology.orchestra.run.smithi088.stdout:Total download size: 48 M 2024-03-29T10:38:36.132 INFO:teuthology.orchestra.run.smithi088.stdout:Installed size: 204 M 2024-03-29T10:38:36.132 INFO:teuthology.orchestra.run.smithi088.stdout:Downloading Packages: 2024-03-29T10:38:36.262 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction test succeeded. 2024-03-29T10:38:36.262 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction 2024-03-29T10:38:36.346 INFO:teuthology.orchestra.run.smithi088.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.2 MB/s | 217 kB 00:00 2024-03-29T10:38:36.446 INFO:teuthology.orchestra.run.smithi088.stdout:(2/6): libcephsqlite-19.0.0-1439.g5dcb190d.el9. 590 kB/s | 163 kB 00:00 2024-03-29T10:38:36.471 INFO:teuthology.orchestra.run.smithi088.stdout:(3/6): socat-1.7.4.1-5.el9.x86_64.rpm 2.3 MB/s | 300 kB 00:00 2024-03-29T10:38:36.505 INFO:teuthology.orchestra.run.smithi088.stdout:(4/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.0 MB/s | 63 kB 00:00 2024-03-29T10:38:36.547 INFO:teuthology.orchestra.run.smithi088.stdout:(5/6): jq-1.6-16.el9.x86_64.rpm 2.4 MB/s | 187 kB 00:00 2024-03-29T10:38:36.829 INFO:teuthology.orchestra.run.smithi196.stdout: Preparing : 1/1 2024-03-29T10:38:36.903 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-29T10:38:36.941 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-29T10:38:36.981 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-29T10:38:37.029 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-29T10:38:37.056 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-29T10:38:40.830 INFO:teuthology.orchestra.run.smithi088.stdout:(6/6): ceph-test-19.0.0-1439.g5dcb190d.el9.x86_ 10 MB/s | 47 MB 00:04 2024-03-29T10:38:40.832 INFO:teuthology.orchestra.run.smithi088.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:38:40.833 INFO:teuthology.orchestra.run.smithi088.stdout:Total 10 MB/s | 48 MB 00:04 2024-03-29T10:38:40.847 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction check 2024-03-29T10:38:40.869 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction check succeeded. 2024-03-29T10:38:40.869 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction test 2024-03-29T10:38:41.249 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction test succeeded. 2024-03-29T10:38:41.250 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction 2024-03-29T10:38:41.539 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-29T10:38:41.623 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-29T10:38:41.847 INFO:teuthology.orchestra.run.smithi088.stdout: Preparing : 1/1 2024-03-29T10:38:41.930 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-29T10:38:41.990 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-29T10:38:42.031 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-29T10:38:42.080 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-29T10:38:42.100 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-29T10:38:44.042 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-29T10:38:44.042 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/6 2024-03-29T10:38:44.042 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/6 2024-03-29T10:38:44.042 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-29T10:38:44.042 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-29T10:38:44.042 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-29T10:38:44.382 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-29T10:38:44.382 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-29T10:38:44.383 INFO:teuthology.orchestra.run.smithi196.stdout:Installed: 2024-03-29T10:38:44.383 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:38:44.383 INFO:teuthology.orchestra.run.smithi196.stdout: jq-1.6-16.el9.x86_64 2024-03-29T10:38:44.383 INFO:teuthology.orchestra.run.smithi196.stdout: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:38:44.383 INFO:teuthology.orchestra.run.smithi196.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-29T10:38:44.383 INFO:teuthology.orchestra.run.smithi196.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-29T10:38:44.383 INFO:teuthology.orchestra.run.smithi196.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-29T10:38:44.383 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-29T10:38:44.383 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-03-29T10:38:44.689 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install ceph 2024-03-29T10:38:45.268 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:00:41 ago on Fri 29 Mar 2024 10:38:04 AM UTC. 2024-03-29T10:38:45.429 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-03-29T10:38:45.432 INFO:teuthology.orchestra.run.smithi196.stdout:========================================================================================= 2024-03-29T10:38:45.432 INFO:teuthology.orchestra.run.smithi196.stdout: Package Arch Version Repository Size 2024-03-29T10:38:45.432 INFO:teuthology.orchestra.run.smithi196.stdout:========================================================================================= 2024-03-29T10:38:45.432 INFO:teuthology.orchestra.run.smithi196.stdout:Installing: 2024-03-29T10:38:45.432 INFO:teuthology.orchestra.run.smithi196.stdout: ceph x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 6.6 k 2024-03-29T10:38:45.432 INFO:teuthology.orchestra.run.smithi196.stdout:Installing dependencies: 2024-03-29T10:38:45.432 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mds x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 2.1 M 2024-03-29T10:38:45.432 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mgr x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.5 M 2024-03-29T10:38:45.432 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 247 k 2024-03-29T10:38:45.432 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mon x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 4.7 M 2024-03-29T10:38:45.432 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-osd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 17 M 2024-03-29T10:38:45.432 INFO:teuthology.orchestra.run.smithi196.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-29T10:38:45.433 INFO:teuthology.orchestra.run.smithi196.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-29T10:38:45.433 INFO:teuthology.orchestra.run.smithi196.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-29T10:38:45.433 INFO:teuthology.orchestra.run.smithi196.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-29T10:38:45.433 INFO:teuthology.orchestra.run.smithi196.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-29T10:38:45.433 INFO:teuthology.orchestra.run.smithi196.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-29T10:38:45.433 INFO:teuthology.orchestra.run.smithi196.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-29T10:38:45.433 INFO:teuthology.orchestra.run.smithi196.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-29T10:38:45.433 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-29T10:38:45.433 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-29T10:38:45.433 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-29T10:38:45.433 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-29T10:38:45.433 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-29T10:38:45.433 INFO:teuthology.orchestra.run.smithi196.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-29T10:38:45.433 INFO:teuthology.orchestra.run.smithi196.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-29T10:38:45.434 INFO:teuthology.orchestra.run.smithi196.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-29T10:38:45.434 INFO:teuthology.orchestra.run.smithi196.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-29T10:38:45.434 INFO:teuthology.orchestra.run.smithi196.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-29T10:38:45.434 INFO:teuthology.orchestra.run.smithi196.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-29T10:38:45.434 INFO:teuthology.orchestra.run.smithi196.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-29T10:38:45.434 INFO:teuthology.orchestra.run.smithi196.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-29T10:38:45.434 INFO:teuthology.orchestra.run.smithi196.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-29T10:38:45.434 INFO:teuthology.orchestra.run.smithi196.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-29T10:38:45.434 INFO:teuthology.orchestra.run.smithi196.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-29T10:38:45.434 INFO:teuthology.orchestra.run.smithi196.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-29T10:38:45.434 INFO:teuthology.orchestra.run.smithi196.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-29T10:38:45.434 INFO:teuthology.orchestra.run.smithi196.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-29T10:38:45.435 INFO:teuthology.orchestra.run.smithi196.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-29T10:38:45.435 INFO:teuthology.orchestra.run.smithi196.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-29T10:38:45.435 INFO:teuthology.orchestra.run.smithi196.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-29T10:38:45.435 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-29T10:38:45.435 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction Summary 2024-03-29T10:38:45.435 INFO:teuthology.orchestra.run.smithi196.stdout:========================================================================================= 2024-03-29T10:38:45.435 INFO:teuthology.orchestra.run.smithi196.stdout:Install 36 Packages 2024-03-29T10:38:45.435 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-29T10:38:45.436 INFO:teuthology.orchestra.run.smithi196.stdout:Total download size: 30 M 2024-03-29T10:38:45.436 INFO:teuthology.orchestra.run.smithi196.stdout:Installed size: 107 M 2024-03-29T10:38:45.436 INFO:teuthology.orchestra.run.smithi196.stdout:Downloading Packages: 2024-03-29T10:38:46.348 INFO:teuthology.orchestra.run.smithi196.stdout:(1/36): ceph-19.0.0-1439.g5dcb190d.el9.x86_64.r 23 kB/s | 6.6 kB 00:00 2024-03-29T10:38:46.566 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-29T10:38:46.648 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-29T10:38:47.273 INFO:teuthology.orchestra.run.smithi196.stdout:(2/36): ceph-mgr-19.0.0-1439.g5dcb190d.el9.x86_ 1.2 MB/s | 1.5 MB 00:01 2024-03-29T10:38:47.440 INFO:teuthology.orchestra.run.smithi196.stdout:(3/36): ceph-mds-19.0.0-1439.g5dcb190d.el9.x86_ 1.5 MB/s | 2.1 MB 00:01 2024-03-29T10:38:47.557 INFO:teuthology.orchestra.run.smithi196.stdout:(4/36): ceph-mgr-modules-core-19.0.0-1439.g5dcb 2.1 MB/s | 247 kB 00:00 2024-03-29T10:38:47.665 INFO:teuthology.orchestra.run.smithi196.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 1.7 MB/s | 187 kB 00:00 2024-03-29T10:38:47.732 INFO:teuthology.orchestra.run.smithi196.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 2.1 MB/s | 144 kB 00:00 2024-03-29T10:38:47.758 INFO:teuthology.orchestra.run.smithi196.stdout:(7/36): python3-markupsafe-1.1.1-12.el9.x86_64. 1.2 MB/s | 32 kB 00:00 2024-03-29T10:38:47.783 INFO:teuthology.orchestra.run.smithi196.stdout:(8/36): python3-pytz-2021.1-5.el9.noarch.rpm 1.9 MB/s | 47 kB 00:00 2024-03-29T10:38:47.808 INFO:teuthology.orchestra.run.smithi196.stdout:(9/36): python3-toml-0.10.2-6.el9.noarch.rpm 1.5 MB/s | 39 kB 00:00 2024-03-29T10:38:47.850 INFO:teuthology.orchestra.run.smithi196.stdout:(10/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 5.6 MB/s | 241 kB 00:00 2024-03-29T10:38:47.926 INFO:teuthology.orchestra.run.smithi196.stdout:(11/36): python3-cryptography-36.0.1-4.el9.x86_ 15 MB/s | 1.1 MB 00:00 2024-03-29T10:38:47.959 INFO:teuthology.orchestra.run.smithi196.stdout:(12/36): python3-ply-3.11-14.el9.noarch.rpm 3.0 MB/s | 103 kB 00:00 2024-03-29T10:38:47.993 INFO:teuthology.orchestra.run.smithi196.stdout:(13/36): python3-pycparser-2.20-6.el9.noarch.rp 3.6 MB/s | 124 kB 00:00 2024-03-29T10:38:48.035 INFO:teuthology.orchestra.run.smithi196.stdout:(14/36): python3-urllib3-1.26.5-5.el9.noarch.rp 4.4 MB/s | 187 kB 00:00 2024-03-29T10:38:48.785 INFO:teuthology.orchestra.run.smithi196.stdout:(15/36): ceph-mon-19.0.0-1439.g5dcb190d.el9.x86 1.9 MB/s | 4.7 MB 00:02 2024-03-29T10:38:48.877 INFO:teuthology.orchestra.run.smithi196.stdout:(16/36): python3-requests-2.25.1-8.el9.noarch.r 148 kB/s | 125 kB 00:00 2024-03-29T10:38:48.950 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-29T10:38:48.950 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/6 2024-03-29T10:38:48.950 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/6 2024-03-29T10:38:48.951 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-29T10:38:48.951 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-29T10:38:48.951 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-29T10:38:49.036 INFO:teuthology.orchestra.run.smithi196.stdout:(17/36): luarocks-3.9.2-1.el9.noarch.rpm 956 kB/s | 151 kB 00:00 2024-03-29T10:38:49.052 INFO:teuthology.orchestra.run.smithi196.stdout:(18/36): lua-devel-5.4.4-4.el9.x86_64.rpm 83 kB/s | 22 kB 00:00 2024-03-29T10:38:49.078 INFO:teuthology.orchestra.run.smithi196.stdout:(19/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 1.0 MB/s | 43 kB 00:00 2024-03-29T10:38:49.161 INFO:teuthology.orchestra.run.smithi196.stdout:(20/36): python3-cherrypy-18.6.1-2.el9.noarch.r 4.2 MB/s | 358 kB 00:00 2024-03-29T10:38:49.332 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-29T10:38:49.332 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-03-29T10:38:49.332 INFO:teuthology.orchestra.run.smithi088.stdout:Installed: 2024-03-29T10:38:49.332 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:38:49.332 INFO:teuthology.orchestra.run.smithi088.stdout: jq-1.6-16.el9.x86_64 2024-03-29T10:38:49.332 INFO:teuthology.orchestra.run.smithi088.stdout: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:38:49.332 INFO:teuthology.orchestra.run.smithi088.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-29T10:38:49.333 INFO:teuthology.orchestra.run.smithi088.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-29T10:38:49.333 INFO:teuthology.orchestra.run.smithi088.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-29T10:38:49.333 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-03-29T10:38:49.333 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2024-03-29T10:38:49.345 INFO:teuthology.orchestra.run.smithi196.stdout:(21/36): ceph-osd-19.0.0-1439.g5dcb190d.el9.x86 8.1 MB/s | 17 MB 00:02 2024-03-29T10:38:49.362 INFO:teuthology.orchestra.run.smithi196.stdout:(22/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 53 kB/s | 11 kB 00:00 2024-03-29T10:38:49.387 INFO:teuthology.orchestra.run.smithi196.stdout:(23/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 513 kB/s | 172 kB 00:00 2024-03-29T10:38:49.413 INFO:teuthology.orchestra.run.smithi196.stdout:(24/36): python3-jaraco-classes-3.2.1-5.el9.noa 264 kB/s | 18 kB 00:00 2024-03-29T10:38:49.438 INFO:teuthology.orchestra.run.smithi196.stdout:(25/36): python3-jaraco-functools-3.5.0-2.el9.n 386 kB/s | 19 kB 00:00 2024-03-29T10:38:49.463 INFO:teuthology.orchestra.run.smithi196.stdout:(26/36): python3-jaraco-collections-3.0.0-8.el9 230 kB/s | 23 kB 00:00 2024-03-29T10:38:49.489 INFO:teuthology.orchestra.run.smithi196.stdout:(27/36): python3-jaraco-text-3.2.0-6.el9.noarch 260 kB/s | 20 kB 00:00 2024-03-29T10:38:49.514 INFO:teuthology.orchestra.run.smithi196.stdout:(28/36): python3-logutils-0.3.5-21.el9.noarch.r 613 kB/s | 46 kB 00:00 2024-03-29T10:38:49.531 INFO:teuthology.orchestra.run.smithi196.stdout:(29/36): python3-more-itertools-8.12.0-2.el9.no 1.1 MB/s | 79 kB 00:00 2024-03-29T10:38:49.556 INFO:teuthology.orchestra.run.smithi196.stdout:(30/36): python3-portend-3.1.0-2.el9.noarch.rpm 392 kB/s | 16 kB 00:00 2024-03-29T10:38:49.573 INFO:teuthology.orchestra.run.smithi196.stdout:(31/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 2.1 MB/s | 90 kB 00:00 2024-03-29T10:38:49.607 INFO:teuthology.orchestra.run.smithi196.stdout:(32/36): python3-tempora-5.0.0-2.el9.noarch.rpm 714 kB/s | 36 kB 00:00 2024-03-29T10:38:49.616 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install ceph 2024-03-29T10:38:49.632 INFO:teuthology.orchestra.run.smithi196.stdout:(33/36): python3-pecan-1.4.2-3.el9.noarch.rpm 1.9 MB/s | 272 kB 00:00 2024-03-29T10:38:49.657 INFO:teuthology.orchestra.run.smithi196.stdout:(34/36): python3-webob-1.8.7-6.el9.noarch.rpm 2.7 MB/s | 230 kB 00:00 2024-03-29T10:38:49.682 INFO:teuthology.orchestra.run.smithi196.stdout:(35/36): python3-zc-lockfile-2.0-10.el9.noarch. 399 kB/s | 20 kB 00:00 2024-03-29T10:38:49.708 INFO:teuthology.orchestra.run.smithi196.stdout:(36/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 4.1 MB/s | 427 kB 00:00 2024-03-29T10:38:49.711 INFO:teuthology.orchestra.run.smithi196.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:38:49.711 INFO:teuthology.orchestra.run.smithi196.stdout:Total 6.9 MB/s | 30 MB 00:04 2024-03-29T10:38:49.734 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction check 2024-03-29T10:38:49.775 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction check succeeded. 2024-03-29T10:38:49.775 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction test 2024-03-29T10:38:50.179 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction test succeeded. 2024-03-29T10:38:50.179 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction 2024-03-29T10:38:50.208 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:00:45 ago on Fri 29 Mar 2024 10:38:05 AM UTC. 2024-03-29T10:38:50.372 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2024-03-29T10:38:50.374 INFO:teuthology.orchestra.run.smithi088.stdout:========================================================================================= 2024-03-29T10:38:50.374 INFO:teuthology.orchestra.run.smithi088.stdout: Package Arch Version Repository Size 2024-03-29T10:38:50.374 INFO:teuthology.orchestra.run.smithi088.stdout:========================================================================================= 2024-03-29T10:38:50.374 INFO:teuthology.orchestra.run.smithi088.stdout:Installing: 2024-03-29T10:38:50.374 INFO:teuthology.orchestra.run.smithi088.stdout: ceph x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 6.6 k 2024-03-29T10:38:50.375 INFO:teuthology.orchestra.run.smithi088.stdout:Installing dependencies: 2024-03-29T10:38:50.375 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mds x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 2.1 M 2024-03-29T10:38:50.375 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mgr x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.5 M 2024-03-29T10:38:50.375 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 247 k 2024-03-29T10:38:50.375 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mon x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 4.7 M 2024-03-29T10:38:50.375 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-osd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 17 M 2024-03-29T10:38:50.375 INFO:teuthology.orchestra.run.smithi088.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-29T10:38:50.375 INFO:teuthology.orchestra.run.smithi088.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-29T10:38:50.375 INFO:teuthology.orchestra.run.smithi088.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-29T10:38:50.375 INFO:teuthology.orchestra.run.smithi088.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-29T10:38:50.375 INFO:teuthology.orchestra.run.smithi088.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-29T10:38:50.376 INFO:teuthology.orchestra.run.smithi088.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-29T10:38:50.376 INFO:teuthology.orchestra.run.smithi088.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-29T10:38:50.376 INFO:teuthology.orchestra.run.smithi088.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-29T10:38:50.376 INFO:teuthology.orchestra.run.smithi088.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-29T10:38:50.376 INFO:teuthology.orchestra.run.smithi088.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-29T10:38:50.376 INFO:teuthology.orchestra.run.smithi088.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-29T10:38:50.376 INFO:teuthology.orchestra.run.smithi088.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-29T10:38:50.376 INFO:teuthology.orchestra.run.smithi088.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-29T10:38:50.376 INFO:teuthology.orchestra.run.smithi088.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-29T10:38:50.376 INFO:teuthology.orchestra.run.smithi088.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-29T10:38:50.376 INFO:teuthology.orchestra.run.smithi088.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-29T10:38:50.376 INFO:teuthology.orchestra.run.smithi088.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-29T10:38:50.377 INFO:teuthology.orchestra.run.smithi088.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-29T10:38:50.377 INFO:teuthology.orchestra.run.smithi088.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-29T10:38:50.377 INFO:teuthology.orchestra.run.smithi088.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-29T10:38:50.377 INFO:teuthology.orchestra.run.smithi088.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-29T10:38:50.377 INFO:teuthology.orchestra.run.smithi088.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-29T10:38:50.377 INFO:teuthology.orchestra.run.smithi088.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-29T10:38:50.377 INFO:teuthology.orchestra.run.smithi088.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-29T10:38:50.377 INFO:teuthology.orchestra.run.smithi088.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-29T10:38:50.377 INFO:teuthology.orchestra.run.smithi088.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-29T10:38:50.377 INFO:teuthology.orchestra.run.smithi088.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-29T10:38:50.377 INFO:teuthology.orchestra.run.smithi088.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-29T10:38:50.377 INFO:teuthology.orchestra.run.smithi088.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-29T10:38:50.378 INFO:teuthology.orchestra.run.smithi088.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-29T10:38:50.378 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-03-29T10:38:50.378 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction Summary 2024-03-29T10:38:50.378 INFO:teuthology.orchestra.run.smithi088.stdout:========================================================================================= 2024-03-29T10:38:50.378 INFO:teuthology.orchestra.run.smithi088.stdout:Install 36 Packages 2024-03-29T10:38:50.378 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-03-29T10:38:50.379 INFO:teuthology.orchestra.run.smithi088.stdout:Total download size: 30 M 2024-03-29T10:38:50.379 INFO:teuthology.orchestra.run.smithi088.stdout:Installed size: 107 M 2024-03-29T10:38:50.379 INFO:teuthology.orchestra.run.smithi088.stdout:Downloading Packages: 2024-03-29T10:38:51.007 INFO:teuthology.orchestra.run.smithi088.stdout:(1/36): ceph-19.0.0-1439.g5dcb190d.el9.x86_64.r 39 kB/s | 6.6 kB 00:00 2024-03-29T10:38:51.265 INFO:teuthology.orchestra.run.smithi088.stdout:(2/36): ceph-mgr-19.0.0-1439.g5dcb190d.el9.x86_ 3.5 MB/s | 1.5 MB 00:00 2024-03-29T10:38:51.516 INFO:teuthology.orchestra.run.smithi196.stdout: Preparing : 1/1 2024-03-29T10:38:51.602 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-29T10:38:51.635 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-29T10:38:51.672 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-29T10:38:51.742 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-29T10:38:51.786 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-29T10:38:51.807 INFO:teuthology.orchestra.run.smithi088.stdout:(3/36): ceph-mon-19.0.0-1439.g5dcb190d.el9.x86_ 5.9 MB/s | 4.7 MB 00:00 2024-03-29T10:38:51.819 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-29T10:38:51.858 INFO:teuthology.orchestra.run.smithi088.stdout:(4/36): ceph-mds-19.0.0-1439.g5dcb190d.el9.x86_ 2.1 MB/s | 2.1 MB 00:01 2024-03-29T10:38:51.883 INFO:teuthology.orchestra.run.smithi088.stdout:(5/36): ceph-mgr-modules-core-19.0.0-1439.g5dcb 3.2 MB/s | 247 kB 00:00 2024-03-29T10:38:51.910 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-29T10:38:51.981 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-29T10:38:52.008 INFO:teuthology.orchestra.run.smithi088.stdout:(6/36): lua-5.4.4-4.el9.x86_64.rpm 1.2 MB/s | 187 kB 00:00 2024-03-29T10:38:52.030 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-29T10:38:52.050 INFO:teuthology.orchestra.run.smithi088.stdout:(7/36): python3-mako-1.1.4-6.el9.noarch.rpm 1.0 MB/s | 144 kB 00:00 2024-03-29T10:38:52.079 INFO:teuthology.orchestra.run.smithi088.stdout:(8/36): python3-markupsafe-1.1.1-12.el9.x86_64. 472 kB/s | 32 kB 00:00 2024-03-29T10:38:52.101 INFO:teuthology.orchestra.run.smithi088.stdout:(9/36): python3-pytz-2021.1-5.el9.noarch.rpm 944 kB/s | 47 kB 00:00 2024-03-29T10:38:52.102 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-29T10:38:52.126 INFO:teuthology.orchestra.run.smithi088.stdout:(10/36): python3-toml-0.10.2-6.el9.noarch.rpm 769 kB/s | 39 kB 00:00 2024-03-29T10:38:52.151 INFO:teuthology.orchestra.run.smithi088.stdout:(11/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.7 MB/s | 241 kB 00:00 2024-03-29T10:38:52.155 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-29T10:38:52.194 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-29T10:38:52.209 INFO:teuthology.orchestra.run.smithi088.stdout:(12/36): python3-cryptography-36.0.1-4.el9.x86_ 14 MB/s | 1.1 MB 00:00 2024-03-29T10:38:52.235 INFO:teuthology.orchestra.run.smithi088.stdout:(13/36): python3-ply-3.11-14.el9.noarch.rpm 1.2 MB/s | 103 kB 00:00 2024-03-29T10:38:52.243 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-29T10:38:52.260 INFO:teuthology.orchestra.run.smithi088.stdout:(14/36): python3-pycparser-2.20-6.el9.noarch.rp 2.4 MB/s | 124 kB 00:00 2024-03-29T10:38:52.294 INFO:teuthology.orchestra.run.smithi088.stdout:(15/36): python3-urllib3-1.26.5-5.el9.noarch.rp 3.1 MB/s | 187 kB 00:00 2024-03-29T10:38:52.298 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-29T10:38:52.394 INFO:teuthology.orchestra.run.smithi088.stdout:(16/36): lua-devel-5.4.4-4.el9.x86_64.rpm 222 kB/s | 22 kB 00:00 2024-03-29T10:38:52.486 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-29T10:38:52.540 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-29T10:38:52.570 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-29T10:38:52.604 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-29T10:38:52.635 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-29T10:38:52.709 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-29T10:38:52.728 INFO:teuthology.orchestra.run.smithi088.stdout:(17/36): ceph-osd-19.0.0-1439.g5dcb190d.el9.x86 11 MB/s | 17 MB 00:01 2024-03-29T10:38:52.745 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-29T10:38:52.778 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-29T10:38:52.818 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-29T10:38:52.853 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-29T10:38:52.878 INFO:teuthology.orchestra.run.smithi088.stdout:(18/36): luarocks-3.9.2-1.el9.noarch.rpm 312 kB/s | 151 kB 00:00 2024-03-29T10:38:52.896 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-29T10:38:52.987 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-29T10:38:53.028 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-29T10:38:53.078 INFO:teuthology.orchestra.run.smithi088.stdout:(19/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 124 kB/s | 43 kB 00:00 2024-03-29T10:38:53.100 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-29T10:38:53.212 INFO:teuthology.orchestra.run.smithi088.stdout:(20/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 516 kB/s | 172 kB 00:00 2024-03-29T10:38:53.237 INFO:teuthology.orchestra.run.smithi088.stdout:(21/36): python3-requests-2.25.1-8.el9.noarch.r 128 kB/s | 125 kB 00:00 2024-03-29T10:38:53.244 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-29T10:38:53.296 INFO:teuthology.orchestra.run.smithi088.stdout:(22/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 128 kB/s | 11 kB 00:00 2024-03-29T10:38:53.369 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-29T10:38:53.396 INFO:teuthology.orchestra.run.smithi088.stdout:(23/36): python3-cherrypy-18.6.1-2.el9.noarch.r 1.1 MB/s | 358 kB 00:00 2024-03-29T10:38:53.421 INFO:teuthology.orchestra.run.smithi088.stdout:(24/36): python3-jaraco-collections-3.0.0-8.el9 231 kB/s | 23 kB 00:00 2024-03-29T10:38:53.463 INFO:teuthology.orchestra.run.smithi088.stdout:(25/36): python3-jaraco-functools-3.5.0-2.el9.n 295 kB/s | 19 kB 00:00 2024-03-29T10:38:53.505 INFO:teuthology.orchestra.run.smithi088.stdout:(26/36): python3-jaraco-classes-3.2.1-5.el9.noa 66 kB/s | 18 kB 00:00 2024-03-29T10:38:53.531 INFO:teuthology.orchestra.run.smithi088.stdout:(27/36): python3-jaraco-text-3.2.0-6.el9.noarch 181 kB/s | 20 kB 00:00 2024-03-29T10:38:53.535 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 31/36 2024-03-29T10:38:53.556 INFO:teuthology.orchestra.run.smithi088.stdout:(28/36): python3-logutils-0.3.5-21.el9.noarch.r 501 kB/s | 46 kB 00:00 2024-03-29T10:38:53.566 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-29T10:38:53.592 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-29T10:38:53.592 INFO:teuthology.orchestra.run.smithi196.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:38:53.592 INFO:teuthology.orchestra.run.smithi196.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-29T10:38:53.592 INFO:teuthology.orchestra.run.smithi196.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-29T10:38:53.592 INFO:teuthology.orchestra.run.smithi196.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-29T10:38:53.592 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-29T10:38:53.623 INFO:teuthology.orchestra.run.smithi088.stdout:(29/36): python3-portend-3.1.0-2.el9.noarch.rpm 246 kB/s | 16 kB 00:00 2024-03-29T10:38:53.665 INFO:teuthology.orchestra.run.smithi088.stdout:(30/36): python3-more-itertools-8.12.0-2.el9.no 496 kB/s | 79 kB 00:00 2024-03-29T10:38:53.698 INFO:teuthology.orchestra.run.smithi088.stdout:(31/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.2 MB/s | 90 kB 00:00 2024-03-29T10:38:53.749 INFO:teuthology.orchestra.run.smithi088.stdout:(32/36): python3-pecan-1.4.2-3.el9.noarch.rpm 1.2 MB/s | 272 kB 00:00 2024-03-29T10:38:53.765 INFO:teuthology.orchestra.run.smithi088.stdout:(33/36): python3-tempora-5.0.0-2.el9.noarch.rpm 355 kB/s | 36 kB 00:00 2024-03-29T10:38:53.799 INFO:teuthology.orchestra.run.smithi088.stdout:(34/36): python3-webob-1.8.7-6.el9.noarch.rpm 2.2 MB/s | 230 kB 00:00 2024-03-29T10:38:53.841 INFO:teuthology.orchestra.run.smithi088.stdout:(35/36): python3-zc-lockfile-2.0-10.el9.noarch. 266 kB/s | 20 kB 00:00 2024-03-29T10:38:53.883 INFO:teuthology.orchestra.run.smithi088.stdout:(36/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 3.1 MB/s | 427 kB 00:00 2024-03-29T10:38:53.889 INFO:teuthology.orchestra.run.smithi088.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:38:53.890 INFO:teuthology.orchestra.run.smithi088.stdout:Total 8.4 MB/s | 30 MB 00:03 2024-03-29T10:38:53.914 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction check 2024-03-29T10:38:53.957 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction check succeeded. 2024-03-29T10:38:53.957 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction test 2024-03-29T10:38:54.373 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction test succeeded. 2024-03-29T10:38:54.373 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction 2024-03-29T10:38:55.025 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-29T10:38:55.052 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-29T10:38:55.052 INFO:teuthology.orchestra.run.smithi196.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:38:55.052 INFO:teuthology.orchestra.run.smithi196.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-29T10:38:55.052 INFO:teuthology.orchestra.run.smithi196.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-29T10:38:55.052 INFO:teuthology.orchestra.run.smithi196.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-29T10:38:55.052 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-29T10:38:55.493 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-29T10:38:55.516 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-29T10:38:55.517 INFO:teuthology.orchestra.run.smithi196.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:38:55.517 INFO:teuthology.orchestra.run.smithi196.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-29T10:38:55.517 INFO:teuthology.orchestra.run.smithi196.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-29T10:38:55.517 INFO:teuthology.orchestra.run.smithi196.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-29T10:38:55.517 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-29T10:38:55.702 INFO:teuthology.orchestra.run.smithi088.stdout: Preparing : 1/1 2024-03-29T10:38:55.727 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-29T10:38:55.750 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-29T10:38:55.750 INFO:teuthology.orchestra.run.smithi196.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:38:55.750 INFO:teuthology.orchestra.run.smithi196.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-29T10:38:55.750 INFO:teuthology.orchestra.run.smithi196.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-29T10:38:55.750 INFO:teuthology.orchestra.run.smithi196.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-29T10:38:55.750 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-29T10:38:55.760 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-29T10:38:55.794 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-29T10:38:55.816 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-29T10:38:55.821 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-29T10:38:55.891 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-29T10:38:55.936 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-29T10:38:55.970 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-29T10:38:56.062 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-29T10:38:56.131 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-29T10:38:56.172 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-29T10:38:56.244 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-29T10:38:56.297 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-29T10:38:56.336 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-29T10:38:56.385 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-29T10:38:56.440 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-29T10:38:56.625 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-29T10:38:56.682 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-29T10:38:56.720 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-29T10:38:56.755 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-29T10:38:56.785 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-29T10:38:56.861 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-29T10:38:56.895 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-29T10:38:56.928 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-29T10:38:56.969 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-29T10:38:57.004 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-29T10:38:57.038 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-29T10:38:57.128 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-29T10:38:57.178 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-29T10:38:57.251 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-29T10:38:57.388 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-29T10:38:57.503 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-29T10:38:57.503 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/36 2024-03-29T10:38:57.503 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/36 2024-03-29T10:38:57.503 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/36 2024-03-29T10:38:57.503 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/36 2024-03-29T10:38:57.503 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/36 2024-03-29T10:38:57.503 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 6/36 2024-03-29T10:38:57.504 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-29T10:38:57.504 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-29T10:38:57.504 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-29T10:38:57.504 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-29T10:38:57.504 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-29T10:38:57.504 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-29T10:38:57.504 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-29T10:38:57.504 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-29T10:38:57.504 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-29T10:38:57.504 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-29T10:38:57.504 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-29T10:38:57.504 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-29T10:38:57.504 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-29T10:38:57.505 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-29T10:38:57.505 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-29T10:38:57.505 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-29T10:38:57.505 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-29T10:38:57.505 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-29T10:38:57.505 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-29T10:38:57.505 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-29T10:38:57.505 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-29T10:38:57.505 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-29T10:38:57.506 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-29T10:38:57.506 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-29T10:38:57.506 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-29T10:38:57.506 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-29T10:38:57.506 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-29T10:38:57.506 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-29T10:38:57.507 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-29T10:38:57.514 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-29T10:38:57.677 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 31/36 2024-03-29T10:38:57.715 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-29T10:38:57.741 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-29T10:38:57.741 INFO:teuthology.orchestra.run.smithi088.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:38:57.741 INFO:teuthology.orchestra.run.smithi088.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-29T10:38:57.742 INFO:teuthology.orchestra.run.smithi088.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-29T10:38:57.742 INFO:teuthology.orchestra.run.smithi088.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-29T10:38:57.742 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-03-29T10:38:58.577 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-29T10:38:58.577 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-29T10:38:58.577 INFO:teuthology.orchestra.run.smithi196.stdout:Installed: 2024-03-29T10:38:58.577 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:38:58.577 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:38:58.578 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:38:58.578 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:38:58.578 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:38:58.578 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:38:58.578 INFO:teuthology.orchestra.run.smithi196.stdout: lua-5.4.4-4.el9.x86_64 2024-03-29T10:38:58.578 INFO:teuthology.orchestra.run.smithi196.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-29T10:38:58.578 INFO:teuthology.orchestra.run.smithi196.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-29T10:38:58.578 INFO:teuthology.orchestra.run.smithi196.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-29T10:38:58.578 INFO:teuthology.orchestra.run.smithi196.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-29T10:38:58.578 INFO:teuthology.orchestra.run.smithi196.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-29T10:38:58.578 INFO:teuthology.orchestra.run.smithi196.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-29T10:38:58.579 INFO:teuthology.orchestra.run.smithi196.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-29T10:38:58.579 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-29T10:38:58.579 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-29T10:38:58.579 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-29T10:38:58.579 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-29T10:38:58.579 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-29T10:38:58.579 INFO:teuthology.orchestra.run.smithi196.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-29T10:38:58.579 INFO:teuthology.orchestra.run.smithi196.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-29T10:38:58.579 INFO:teuthology.orchestra.run.smithi196.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-29T10:38:58.579 INFO:teuthology.orchestra.run.smithi196.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-29T10:38:58.579 INFO:teuthology.orchestra.run.smithi196.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-29T10:38:58.579 INFO:teuthology.orchestra.run.smithi196.stdout: python3-ply-3.11-14.el9.noarch 2024-03-29T10:38:58.579 INFO:teuthology.orchestra.run.smithi196.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-29T10:38:58.579 INFO:teuthology.orchestra.run.smithi196.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-29T10:38:58.580 INFO:teuthology.orchestra.run.smithi196.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-29T10:38:58.580 INFO:teuthology.orchestra.run.smithi196.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-29T10:38:58.580 INFO:teuthology.orchestra.run.smithi196.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-29T10:38:58.580 INFO:teuthology.orchestra.run.smithi196.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-29T10:38:58.580 INFO:teuthology.orchestra.run.smithi196.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-29T10:38:58.580 INFO:teuthology.orchestra.run.smithi196.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-29T10:38:58.580 INFO:teuthology.orchestra.run.smithi196.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-29T10:38:58.580 INFO:teuthology.orchestra.run.smithi196.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-29T10:38:58.580 INFO:teuthology.orchestra.run.smithi196.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-29T10:38:58.580 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-29T10:38:58.580 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-03-29T10:38:58.913 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install ceph-base 2024-03-29T10:38:59.175 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-29T10:38:59.202 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-29T10:38:59.202 INFO:teuthology.orchestra.run.smithi088.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:38:59.202 INFO:teuthology.orchestra.run.smithi088.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-29T10:38:59.202 INFO:teuthology.orchestra.run.smithi088.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-29T10:38:59.202 INFO:teuthology.orchestra.run.smithi088.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-29T10:38:59.202 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-03-29T10:38:59.480 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:00:55 ago on Fri 29 Mar 2024 10:38:04 AM UTC. 2024-03-29T10:38:59.578 INFO:teuthology.orchestra.run.smithi196.stdout:Package ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:38:59.630 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-03-29T10:38:59.631 INFO:teuthology.orchestra.run.smithi196.stdout:Nothing to do. 2024-03-29T10:38:59.631 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-03-29T10:38:59.642 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-29T10:38:59.665 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-29T10:38:59.665 INFO:teuthology.orchestra.run.smithi088.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:38:59.665 INFO:teuthology.orchestra.run.smithi088.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-29T10:38:59.665 INFO:teuthology.orchestra.run.smithi088.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-29T10:38:59.666 INFO:teuthology.orchestra.run.smithi088.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-29T10:38:59.666 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-03-29T10:38:59.695 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install cephadm 2024-03-29T10:38:59.884 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-29T10:38:59.908 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-29T10:38:59.908 INFO:teuthology.orchestra.run.smithi088.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:38:59.908 INFO:teuthology.orchestra.run.smithi088.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-29T10:38:59.908 INFO:teuthology.orchestra.run.smithi088.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-29T10:38:59.908 INFO:teuthology.orchestra.run.smithi088.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-29T10:38:59.908 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-03-29T10:38:59.975 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-29T10:39:00.259 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:00:56 ago on Fri 29 Mar 2024 10:38:04 AM UTC. 2024-03-29T10:39:00.410 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-03-29T10:39:00.410 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-29T10:39:00.410 INFO:teuthology.orchestra.run.smithi196.stdout: Package Arch Version Repository Size 2024-03-29T10:39:00.410 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-29T10:39:00.411 INFO:teuthology.orchestra.run.smithi196.stdout:Installing: 2024-03-29T10:39:00.411 INFO:teuthology.orchestra.run.smithi196.stdout: cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 747 k 2024-03-29T10:39:00.411 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-29T10:39:00.411 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction Summary 2024-03-29T10:39:00.411 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-29T10:39:00.411 INFO:teuthology.orchestra.run.smithi196.stdout:Install 1 Package 2024-03-29T10:39:00.411 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-29T10:39:00.412 INFO:teuthology.orchestra.run.smithi196.stdout:Total download size: 747 k 2024-03-29T10:39:00.412 INFO:teuthology.orchestra.run.smithi196.stdout:Installed size: 754 k 2024-03-29T10:39:00.412 INFO:teuthology.orchestra.run.smithi196.stdout:Downloading Packages: 2024-03-29T10:39:00.834 INFO:teuthology.orchestra.run.smithi196.stdout:cephadm-19.0.0-1439.g5dcb190d.el9.noarch.rpm 1.7 MB/s | 747 kB 00:00 2024-03-29T10:39:00.834 INFO:teuthology.orchestra.run.smithi196.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:39:00.835 INFO:teuthology.orchestra.run.smithi196.stdout:Total 1.7 MB/s | 747 kB 00:00 2024-03-29T10:39:00.835 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction check 2024-03-29T10:39:00.839 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction check succeeded. 2024-03-29T10:39:00.840 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction test 2024-03-29T10:39:00.845 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction test succeeded. 2024-03-29T10:39:00.846 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction 2024-03-29T10:39:00.918 INFO:teuthology.orchestra.run.smithi196.stdout: Preparing : 1/1 2024-03-29T10:39:01.360 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T10:39:01.440 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T10:39:01.669 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-29T10:39:01.670 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/36 2024-03-29T10:39:01.670 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/36 2024-03-29T10:39:01.670 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/36 2024-03-29T10:39:01.670 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/36 2024-03-29T10:39:01.670 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/36 2024-03-29T10:39:01.670 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 6/36 2024-03-29T10:39:01.670 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-29T10:39:01.670 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-29T10:39:01.670 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-29T10:39:01.670 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-29T10:39:01.670 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-29T10:39:01.670 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-29T10:39:01.671 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-29T10:39:01.671 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-29T10:39:01.671 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-29T10:39:01.671 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-29T10:39:01.671 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-29T10:39:01.671 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-29T10:39:01.671 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-29T10:39:01.671 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-29T10:39:01.671 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-29T10:39:01.671 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-29T10:39:01.671 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-29T10:39:01.671 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-29T10:39:01.671 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-29T10:39:01.672 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-29T10:39:01.672 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-29T10:39:01.672 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-29T10:39:01.672 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-29T10:39:01.672 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-29T10:39:01.672 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-29T10:39:01.672 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-29T10:39:01.672 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-29T10:39:01.672 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-29T10:39:01.672 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-29T10:39:01.797 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T10:39:02.047 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T10:39:02.047 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-29T10:39:02.047 INFO:teuthology.orchestra.run.smithi196.stdout:Installed: 2024-03-29T10:39:02.047 INFO:teuthology.orchestra.run.smithi196.stdout: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:39:02.047 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-29T10:39:02.048 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-03-29T10:39:02.241 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install ceph-immutable-object-cache 2024-03-29T10:39:02.680 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-29T10:39:02.680 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-03-29T10:39:02.680 INFO:teuthology.orchestra.run.smithi088.stdout:Installed: 2024-03-29T10:39:02.680 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:39:02.680 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:39:02.680 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:39:02.680 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:39:02.680 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:39:02.680 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:39:02.680 INFO:teuthology.orchestra.run.smithi088.stdout: lua-5.4.4-4.el9.x86_64 2024-03-29T10:39:02.680 INFO:teuthology.orchestra.run.smithi088.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-29T10:39:02.680 INFO:teuthology.orchestra.run.smithi088.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-29T10:39:02.681 INFO:teuthology.orchestra.run.smithi088.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-29T10:39:02.681 INFO:teuthology.orchestra.run.smithi088.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-29T10:39:02.681 INFO:teuthology.orchestra.run.smithi088.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-29T10:39:02.681 INFO:teuthology.orchestra.run.smithi088.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-29T10:39:02.681 INFO:teuthology.orchestra.run.smithi088.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-29T10:39:02.681 INFO:teuthology.orchestra.run.smithi088.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-29T10:39:02.681 INFO:teuthology.orchestra.run.smithi088.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-29T10:39:02.681 INFO:teuthology.orchestra.run.smithi088.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-29T10:39:02.681 INFO:teuthology.orchestra.run.smithi088.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-29T10:39:02.681 INFO:teuthology.orchestra.run.smithi088.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-29T10:39:02.681 INFO:teuthology.orchestra.run.smithi088.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-29T10:39:02.681 INFO:teuthology.orchestra.run.smithi088.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-29T10:39:02.681 INFO:teuthology.orchestra.run.smithi088.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-29T10:39:02.681 INFO:teuthology.orchestra.run.smithi088.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-29T10:39:02.682 INFO:teuthology.orchestra.run.smithi088.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-29T10:39:02.682 INFO:teuthology.orchestra.run.smithi088.stdout: python3-ply-3.11-14.el9.noarch 2024-03-29T10:39:02.682 INFO:teuthology.orchestra.run.smithi088.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-29T10:39:02.682 INFO:teuthology.orchestra.run.smithi088.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-29T10:39:02.682 INFO:teuthology.orchestra.run.smithi088.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-29T10:39:02.682 INFO:teuthology.orchestra.run.smithi088.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-29T10:39:02.682 INFO:teuthology.orchestra.run.smithi088.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-29T10:39:02.682 INFO:teuthology.orchestra.run.smithi088.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-29T10:39:02.682 INFO:teuthology.orchestra.run.smithi088.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-29T10:39:02.682 INFO:teuthology.orchestra.run.smithi088.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-29T10:39:02.682 INFO:teuthology.orchestra.run.smithi088.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-29T10:39:02.682 INFO:teuthology.orchestra.run.smithi088.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-29T10:39:02.682 INFO:teuthology.orchestra.run.smithi088.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-29T10:39:02.682 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-03-29T10:39:02.682 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2024-03-29T10:39:02.816 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:00:58 ago on Fri 29 Mar 2024 10:38:04 AM UTC. 2024-03-29T10:39:02.973 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-03-29T10:39:02.973 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-29T10:39:02.974 INFO:teuthology.orchestra.run.smithi196.stdout: Package Arch Version Repo Size 2024-03-29T10:39:02.974 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-29T10:39:02.974 INFO:teuthology.orchestra.run.smithi196.stdout:Installing: 2024-03-29T10:39:02.974 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 145 k 2024-03-29T10:39:02.974 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-29T10:39:02.974 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction Summary 2024-03-29T10:39:02.974 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-29T10:39:02.974 INFO:teuthology.orchestra.run.smithi196.stdout:Install 1 Package 2024-03-29T10:39:02.974 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-29T10:39:02.974 INFO:teuthology.orchestra.run.smithi196.stdout:Total download size: 145 k 2024-03-29T10:39:02.974 INFO:teuthology.orchestra.run.smithi196.stdout:Installed size: 436 k 2024-03-29T10:39:02.974 INFO:teuthology.orchestra.run.smithi196.stdout:Downloading Packages: 2024-03-29T10:39:03.067 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install ceph-base 2024-03-29T10:39:03.333 INFO:teuthology.orchestra.run.smithi196.stdout:ceph-immutable-object-cache-19.0.0-1439.g5dcb19 405 kB/s | 145 kB 00:00 2024-03-29T10:39:03.334 INFO:teuthology.orchestra.run.smithi196.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:39:03.334 INFO:teuthology.orchestra.run.smithi196.stdout:Total 403 kB/s | 145 kB 00:00 2024-03-29T10:39:03.335 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction check 2024-03-29T10:39:03.342 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction check succeeded. 2024-03-29T10:39:03.342 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction test 2024-03-29T10:39:03.395 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction test succeeded. 2024-03-29T10:39:03.395 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction 2024-03-29T10:39:03.540 INFO:teuthology.orchestra.run.smithi196.stdout: Preparing : 1/1 2024-03-29T10:39:03.593 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T10:39:03.617 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T10:39:03.618 INFO:teuthology.orchestra.run.smithi196.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:39:03.618 INFO:teuthology.orchestra.run.smithi196.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-29T10:39:03.618 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-29T10:39:03.652 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:00:58 ago on Fri 29 Mar 2024 10:38:05 AM UTC. 2024-03-29T10:39:03.749 INFO:teuthology.orchestra.run.smithi088.stdout:Package ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:39:03.801 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2024-03-29T10:39:03.803 INFO:teuthology.orchestra.run.smithi088.stdout:Nothing to do. 2024-03-29T10:39:03.803 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2024-03-29T10:39:03.859 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install cephadm 2024-03-29T10:39:04.425 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:00:59 ago on Fri 29 Mar 2024 10:38:05 AM UTC. 2024-03-29T10:39:04.573 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T10:39:04.573 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-29T10:39:04.574 INFO:teuthology.orchestra.run.smithi196.stdout:Installed: 2024-03-29T10:39:04.574 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:39:04.574 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-29T10:39:04.574 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-03-29T10:39:04.575 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2024-03-29T10:39:04.578 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-03-29T10:39:04.578 INFO:teuthology.orchestra.run.smithi088.stdout: Package Arch Version Repository Size 2024-03-29T10:39:04.578 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-03-29T10:39:04.578 INFO:teuthology.orchestra.run.smithi088.stdout:Installing: 2024-03-29T10:39:04.579 INFO:teuthology.orchestra.run.smithi088.stdout: cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 747 k 2024-03-29T10:39:04.579 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-03-29T10:39:04.579 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction Summary 2024-03-29T10:39:04.579 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-03-29T10:39:04.579 INFO:teuthology.orchestra.run.smithi088.stdout:Install 1 Package 2024-03-29T10:39:04.579 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-03-29T10:39:04.579 INFO:teuthology.orchestra.run.smithi088.stdout:Total download size: 747 k 2024-03-29T10:39:04.579 INFO:teuthology.orchestra.run.smithi088.stdout:Installed size: 754 k 2024-03-29T10:39:04.579 INFO:teuthology.orchestra.run.smithi088.stdout:Downloading Packages: 2024-03-29T10:39:04.837 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install ceph-mgr 2024-03-29T10:39:05.091 INFO:teuthology.orchestra.run.smithi088.stdout:cephadm-19.0.0-1439.g5dcb190d.el9.noarch.rpm 1.4 MB/s | 747 kB 00:00 2024-03-29T10:39:05.092 INFO:teuthology.orchestra.run.smithi088.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:39:05.092 INFO:teuthology.orchestra.run.smithi088.stdout:Total 1.4 MB/s | 747 kB 00:00 2024-03-29T10:39:05.092 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction check 2024-03-29T10:39:05.096 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction check succeeded. 2024-03-29T10:39:05.097 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction test 2024-03-29T10:39:05.102 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction test succeeded. 2024-03-29T10:39:05.103 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction 2024-03-29T10:39:05.173 INFO:teuthology.orchestra.run.smithi088.stdout: Preparing : 1/1 2024-03-29T10:39:05.409 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:01:01 ago on Fri 29 Mar 2024 10:38:04 AM UTC. 2024-03-29T10:39:05.508 INFO:teuthology.orchestra.run.smithi196.stdout:Package ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:39:05.560 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-03-29T10:39:05.561 INFO:teuthology.orchestra.run.smithi196.stdout:Nothing to do. 2024-03-29T10:39:05.561 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-03-29T10:39:05.590 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T10:39:05.623 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install ceph-mgr-dashboard 2024-03-29T10:39:05.662 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T10:39:06.098 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T10:39:06.188 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:01:02 ago on Fri 29 Mar 2024 10:38:04 AM UTC. 2024-03-29T10:39:06.340 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-03-29T10:39:06.341 INFO:teuthology.orchestra.run.smithi196.stdout:====================================================================================== 2024-03-29T10:39:06.341 INFO:teuthology.orchestra.run.smithi196.stdout: Package Arch Version Repository Size 2024-03-29T10:39:06.341 INFO:teuthology.orchestra.run.smithi196.stdout:====================================================================================== 2024-03-29T10:39:06.341 INFO:teuthology.orchestra.run.smithi196.stdout:Installing: 2024-03-29T10:39:06.341 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 1.8 M 2024-03-29T10:39:06.341 INFO:teuthology.orchestra.run.smithi196.stdout:Installing dependencies: 2024-03-29T10:39:06.341 INFO:teuthology.orchestra.run.smithi196.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-29T10:39:06.341 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 31 k 2024-03-29T10:39:06.341 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 15 k 2024-03-29T10:39:06.341 INFO:teuthology.orchestra.run.smithi196.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-29T10:39:06.342 INFO:teuthology.orchestra.run.smithi196.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-29T10:39:06.342 INFO:teuthology.orchestra.run.smithi196.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-29T10:39:06.342 INFO:teuthology.orchestra.run.smithi196.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-29T10:39:06.342 INFO:teuthology.orchestra.run.smithi196.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-29T10:39:06.342 INFO:teuthology.orchestra.run.smithi196.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-29T10:39:06.342 INFO:teuthology.orchestra.run.smithi196.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-29T10:39:06.342 INFO:teuthology.orchestra.run.smithi196.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-29T10:39:06.342 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-29T10:39:06.342 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction Summary 2024-03-29T10:39:06.342 INFO:teuthology.orchestra.run.smithi196.stdout:====================================================================================== 2024-03-29T10:39:06.342 INFO:teuthology.orchestra.run.smithi196.stdout:Install 12 Packages 2024-03-29T10:39:06.342 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-29T10:39:06.343 INFO:teuthology.orchestra.run.smithi196.stdout:Total download size: 6.9 M 2024-03-29T10:39:06.343 INFO:teuthology.orchestra.run.smithi196.stdout:Installed size: 26 M 2024-03-29T10:39:06.343 INFO:teuthology.orchestra.run.smithi196.stdout:Downloading Packages: 2024-03-29T10:39:06.414 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T10:39:06.414 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-03-29T10:39:06.414 INFO:teuthology.orchestra.run.smithi088.stdout:Installed: 2024-03-29T10:39:06.414 INFO:teuthology.orchestra.run.smithi088.stdout: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:39:06.414 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-03-29T10:39:06.415 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2024-03-29T10:39:06.654 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install ceph-immutable-object-cache 2024-03-29T10:39:06.821 INFO:teuthology.orchestra.run.smithi196.stdout:(1/12): ceph-grafana-dashboards-19.0.0-1439.g5d 127 kB/s | 31 kB 00:00 2024-03-29T10:39:06.846 INFO:teuthology.orchestra.run.smithi196.stdout:(2/12): ceph-prometheus-alerts-19.0.0-1439.g5dc 56 kB/s | 15 kB 00:00 2024-03-29T10:39:06.997 INFO:teuthology.orchestra.run.smithi196.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 1.8 MB/s | 238 kB 00:00 2024-03-29T10:39:07.038 INFO:teuthology.orchestra.run.smithi196.stdout:(4/12): protobuf-3.14.0-13.el9.x86_64.rpm 4.6 MB/s | 1.0 MB 00:00 2024-03-29T10:39:07.097 INFO:teuthology.orchestra.run.smithi196.stdout:(5/12): ceph-mgr-dashboard-19.0.0-1439.g5dcb190 3.4 MB/s | 1.8 MB 00:00 2024-03-29T10:39:07.181 INFO:teuthology.orchestra.run.smithi196.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 233 kB/s | 19 kB 00:00 2024-03-29T10:39:07.217 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:01:02 ago on Fri 29 Mar 2024 10:38:05 AM UTC. 2024-03-29T10:39:07.256 INFO:teuthology.orchestra.run.smithi196.stdout:(7/12): protobuf-compiler-3.14.0-13.el9.x86_64. 3.3 MB/s | 863 kB 00:00 2024-03-29T10:39:07.290 INFO:teuthology.orchestra.run.smithi196.stdout:(8/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 2.1 MB/s | 548 kB 00:00 2024-03-29T10:39:07.332 INFO:teuthology.orchestra.run.smithi196.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 737 kB/s | 31 kB 00:00 2024-03-29T10:39:07.368 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2024-03-29T10:39:07.369 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-03-29T10:39:07.369 INFO:teuthology.orchestra.run.smithi088.stdout: Package Arch Version Repo Size 2024-03-29T10:39:07.369 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-03-29T10:39:07.369 INFO:teuthology.orchestra.run.smithi088.stdout:Installing: 2024-03-29T10:39:07.369 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 145 k 2024-03-29T10:39:07.369 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-03-29T10:39:07.369 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction Summary 2024-03-29T10:39:07.369 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-03-29T10:39:07.369 INFO:teuthology.orchestra.run.smithi088.stdout:Install 1 Package 2024-03-29T10:39:07.369 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-03-29T10:39:07.369 INFO:teuthology.orchestra.run.smithi088.stdout:Total download size: 145 k 2024-03-29T10:39:07.369 INFO:teuthology.orchestra.run.smithi088.stdout:Installed size: 436 k 2024-03-29T10:39:07.370 INFO:teuthology.orchestra.run.smithi088.stdout:Downloading Packages: 2024-03-29T10:39:07.382 INFO:teuthology.orchestra.run.smithi196.stdout:(10/12): python3-routes-2.5.1-5.el9.noarch.rpm 3.7 MB/s | 188 kB 00:00 2024-03-29T10:39:07.407 INFO:teuthology.orchestra.run.smithi196.stdout:(11/12): python3-grpcio-tools-1.46.7-10.el9.x86 958 kB/s | 144 kB 00:00 2024-03-29T10:39:07.457 INFO:teuthology.orchestra.run.smithi196.stdout:(12/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 7.4 MB/s | 2.0 MB 00:00 2024-03-29T10:39:07.460 INFO:teuthology.orchestra.run.smithi196.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:39:07.460 INFO:teuthology.orchestra.run.smithi196.stdout:Total 6.1 MB/s | 6.9 MB 00:01 2024-03-29T10:39:07.478 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction check 2024-03-29T10:39:07.495 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction check succeeded. 2024-03-29T10:39:07.495 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction test 2024-03-29T10:39:07.672 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction test succeeded. 2024-03-29T10:39:07.672 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction 2024-03-29T10:39:07.688 INFO:teuthology.orchestra.run.smithi088.stdout:ceph-immutable-object-cache-19.0.0-1439.g5dcb19 457 kB/s | 145 kB 00:00 2024-03-29T10:39:07.688 INFO:teuthology.orchestra.run.smithi088.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:39:07.689 INFO:teuthology.orchestra.run.smithi088.stdout:Total 455 kB/s | 145 kB 00:00 2024-03-29T10:39:07.689 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction check 2024-03-29T10:39:07.696 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction check succeeded. 2024-03-29T10:39:07.696 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction test 2024-03-29T10:39:07.748 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction test succeeded. 2024-03-29T10:39:07.749 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction 2024-03-29T10:39:07.886 INFO:teuthology.orchestra.run.smithi088.stdout: Preparing : 1/1 2024-03-29T10:39:07.964 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T10:39:07.988 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T10:39:07.988 INFO:teuthology.orchestra.run.smithi088.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:39:07.988 INFO:teuthology.orchestra.run.smithi088.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-29T10:39:07.988 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-03-29T10:39:08.182 INFO:teuthology.orchestra.run.smithi196.stdout: Preparing : 1/1 2024-03-29T10:39:08.249 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-29T10:39:08.285 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-29T10:39:08.332 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-29T10:39:08.362 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-29T10:39:08.478 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-29T10:39:08.606 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-29T10:39:08.699 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-29T10:39:08.750 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-29T10:39:08.779 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-29T10:39:08.810 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 10/12 2024-03-29T10:39:08.935 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T10:39:08.935 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-03-29T10:39:08.935 INFO:teuthology.orchestra.run.smithi088.stdout:Installed: 2024-03-29T10:39:08.935 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:39:08.935 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-03-29T10:39:08.935 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2024-03-29T10:39:09.054 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 11/12 2024-03-29T10:39:09.094 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-29T10:39:09.215 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install ceph-mgr 2024-03-29T10:39:09.786 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:01:04 ago on Fri 29 Mar 2024 10:38:05 AM UTC. 2024-03-29T10:39:09.884 INFO:teuthology.orchestra.run.smithi088.stdout:Package ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:39:09.935 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-29T10:39:09.935 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 1/12 2024-03-29T10:39:09.935 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 2/12 2024-03-29T10:39:09.935 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 3/12 2024-03-29T10:39:09.935 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-29T10:39:09.935 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-29T10:39:09.935 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-29T10:39:09.936 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-29T10:39:09.936 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-29T10:39:09.936 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-29T10:39:09.936 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-29T10:39:09.936 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-29T10:39:09.937 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2024-03-29T10:39:09.938 INFO:teuthology.orchestra.run.smithi088.stdout:Nothing to do. 2024-03-29T10:39:09.938 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2024-03-29T10:39:10.004 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install ceph-mgr-dashboard 2024-03-29T10:39:10.305 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-29T10:39:10.306 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-29T10:39:10.306 INFO:teuthology.orchestra.run.smithi196.stdout:Installed: 2024-03-29T10:39:10.306 INFO:teuthology.orchestra.run.smithi196.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-29T10:39:10.306 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:39:10.306 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:39:10.306 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:39:10.306 INFO:teuthology.orchestra.run.smithi196.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-29T10:39:10.306 INFO:teuthology.orchestra.run.smithi196.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-29T10:39:10.307 INFO:teuthology.orchestra.run.smithi196.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-29T10:39:10.307 INFO:teuthology.orchestra.run.smithi196.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-29T10:39:10.307 INFO:teuthology.orchestra.run.smithi196.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-29T10:39:10.307 INFO:teuthology.orchestra.run.smithi196.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-29T10:39:10.307 INFO:teuthology.orchestra.run.smithi196.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-29T10:39:10.307 INFO:teuthology.orchestra.run.smithi196.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-29T10:39:10.307 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-29T10:39:10.307 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-03-29T10:39:10.565 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:01:05 ago on Fri 29 Mar 2024 10:38:05 AM UTC. 2024-03-29T10:39:10.570 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-29T10:39:10.719 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2024-03-29T10:39:10.719 INFO:teuthology.orchestra.run.smithi088.stdout:====================================================================================== 2024-03-29T10:39:10.719 INFO:teuthology.orchestra.run.smithi088.stdout: Package Arch Version Repository Size 2024-03-29T10:39:10.719 INFO:teuthology.orchestra.run.smithi088.stdout:====================================================================================== 2024-03-29T10:39:10.719 INFO:teuthology.orchestra.run.smithi088.stdout:Installing: 2024-03-29T10:39:10.719 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 1.8 M 2024-03-29T10:39:10.720 INFO:teuthology.orchestra.run.smithi088.stdout:Installing dependencies: 2024-03-29T10:39:10.720 INFO:teuthology.orchestra.run.smithi088.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-29T10:39:10.720 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 31 k 2024-03-29T10:39:10.720 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 15 k 2024-03-29T10:39:10.720 INFO:teuthology.orchestra.run.smithi088.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-29T10:39:10.720 INFO:teuthology.orchestra.run.smithi088.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-29T10:39:10.720 INFO:teuthology.orchestra.run.smithi088.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-29T10:39:10.720 INFO:teuthology.orchestra.run.smithi088.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-29T10:39:10.720 INFO:teuthology.orchestra.run.smithi088.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-29T10:39:10.720 INFO:teuthology.orchestra.run.smithi088.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-29T10:39:10.720 INFO:teuthology.orchestra.run.smithi088.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-29T10:39:10.720 INFO:teuthology.orchestra.run.smithi088.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-29T10:39:10.720 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-03-29T10:39:10.720 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction Summary 2024-03-29T10:39:10.721 INFO:teuthology.orchestra.run.smithi088.stdout:====================================================================================== 2024-03-29T10:39:10.721 INFO:teuthology.orchestra.run.smithi088.stdout:Install 12 Packages 2024-03-29T10:39:10.721 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-03-29T10:39:10.721 INFO:teuthology.orchestra.run.smithi088.stdout:Total download size: 6.9 M 2024-03-29T10:39:10.721 INFO:teuthology.orchestra.run.smithi088.stdout:Installed size: 26 M 2024-03-29T10:39:10.721 INFO:teuthology.orchestra.run.smithi088.stdout:Downloading Packages: 2024-03-29T10:39:11.132 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:01:07 ago on Fri 29 Mar 2024 10:38:04 AM UTC. 2024-03-29T10:39:11.284 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-03-29T10:39:11.285 INFO:teuthology.orchestra.run.smithi196.stdout:============================================================================================ 2024-03-29T10:39:11.285 INFO:teuthology.orchestra.run.smithi196.stdout: Package Arch Version Repository Size 2024-03-29T10:39:11.285 INFO:teuthology.orchestra.run.smithi196.stdout:============================================================================================ 2024-03-29T10:39:11.285 INFO:teuthology.orchestra.run.smithi196.stdout:Installing: 2024-03-29T10:39:11.285 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 7.4 M 2024-03-29T10:39:11.285 INFO:teuthology.orchestra.run.smithi196.stdout:Installing dependencies: 2024-03-29T10:39:11.285 INFO:teuthology.orchestra.run.smithi196.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-29T10:39:11.285 INFO:teuthology.orchestra.run.smithi196.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-29T10:39:11.285 INFO:teuthology.orchestra.run.smithi196.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-29T10:39:11.285 INFO:teuthology.orchestra.run.smithi196.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-29T10:39:11.286 INFO:teuthology.orchestra.run.smithi196.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-29T10:39:11.286 INFO:teuthology.orchestra.run.smithi196.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-29T10:39:11.286 INFO:teuthology.orchestra.run.smithi196.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-29T10:39:11.286 INFO:teuthology.orchestra.run.smithi196.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-29T10:39:11.286 INFO:teuthology.orchestra.run.smithi196.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-29T10:39:11.286 INFO:teuthology.orchestra.run.smithi196.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-29T10:39:11.286 INFO:teuthology.orchestra.run.smithi196.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-29T10:39:11.286 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-29T10:39:11.286 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction Summary 2024-03-29T10:39:11.286 INFO:teuthology.orchestra.run.smithi196.stdout:============================================================================================ 2024-03-29T10:39:11.286 INFO:teuthology.orchestra.run.smithi196.stdout:Install 12 Packages 2024-03-29T10:39:11.286 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-29T10:39:11.287 INFO:teuthology.orchestra.run.smithi196.stdout:Total download size: 37 M 2024-03-29T10:39:11.287 INFO:teuthology.orchestra.run.smithi196.stdout:Installed size: 209 M 2024-03-29T10:39:11.287 INFO:teuthology.orchestra.run.smithi196.stdout:Downloading Packages: 2024-03-29T10:39:11.300 INFO:teuthology.orchestra.run.smithi088.stdout:(1/12): ceph-grafana-dashboards-19.0.0-1439.g5d 174 kB/s | 31 kB 00:00 2024-03-29T10:39:11.416 INFO:teuthology.orchestra.run.smithi088.stdout:(2/12): ceph-prometheus-alerts-19.0.0-1439.g5dc 52 kB/s | 15 kB 00:00 2024-03-29T10:39:11.550 INFO:teuthology.orchestra.run.smithi088.stdout:(3/12): protobuf-3.14.0-13.el9.x86_64.rpm 4.0 MB/s | 1.0 MB 00:00 2024-03-29T10:39:11.556 INFO:teuthology.orchestra.run.smithi196.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 185 kB/s | 32 kB 00:00 2024-03-29T10:39:11.575 INFO:teuthology.orchestra.run.smithi088.stdout:(4/12): python3-protobuf-3.14.0-13.el9.noarch.r 1.5 MB/s | 238 kB 00:00 2024-03-29T10:39:11.590 INFO:teuthology.orchestra.run.smithi196.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 535 kB/s | 17 kB 00:00 2024-03-29T10:39:11.623 INFO:teuthology.orchestra.run.smithi196.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 1.0 MB/s | 34 kB 00:00 2024-03-29T10:39:11.682 INFO:teuthology.orchestra.run.smithi196.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 10 MB/s | 3.0 MB 00:00 2024-03-29T10:39:11.692 INFO:teuthology.orchestra.run.smithi088.stdout:(5/12): protobuf-compiler-3.14.0-13.el9.x86_64. 5.9 MB/s | 863 kB 00:00 2024-03-29T10:39:11.776 INFO:teuthology.orchestra.run.smithi088.stdout:(6/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 2.7 MB/s | 548 kB 00:00 2024-03-29T10:39:11.801 INFO:teuthology.orchestra.run.smithi088.stdout:(7/12): grpc-data-1.46.7-10.el9.noarch.rpm 179 kB/s | 19 kB 00:00 2024-03-29T10:39:11.849 INFO:teuthology.orchestra.run.smithi196.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 21 MB/s | 4.7 MB 00:00 2024-03-29T10:39:11.893 INFO:teuthology.orchestra.run.smithi088.stdout:(8/12): python3-grpcio-1.46.7-10.el9.x86_64.rpm 17 MB/s | 2.0 MB 00:00 2024-03-29T10:39:11.916 INFO:teuthology.orchestra.run.smithi196.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 21 MB/s | 4.8 MB 00:00 2024-03-29T10:39:11.918 INFO:teuthology.orchestra.run.smithi088.stdout:(9/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 1.2 MB/s | 144 kB 00:00 2024-03-29T10:39:11.941 INFO:teuthology.orchestra.run.smithi196.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.4 MB/s | 217 kB 00:00 2024-03-29T10:39:11.947 INFO:teuthology.orchestra.run.smithi088.stdout:(10/12): python3-repoze-lru-0.7-16.el9.noarch.r 610 kB/s | 31 kB 00:00 2024-03-29T10:39:11.994 INFO:teuthology.orchestra.run.smithi088.stdout:(11/12): python3-routes-2.5.1-5.el9.noarch.rpm 2.4 MB/s | 188 kB 00:00 2024-03-29T10:39:12.086 INFO:teuthology.orchestra.run.smithi088.stdout:(12/12): ceph-mgr-dashboard-19.0.0-1439.g5dcb19 1.8 MB/s | 1.8 MB 00:00 2024-03-29T10:39:12.088 INFO:teuthology.orchestra.run.smithi088.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:39:12.088 INFO:teuthology.orchestra.run.smithi088.stdout:Total 5.0 MB/s | 6.9 MB 00:01 2024-03-29T10:39:12.106 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction check 2024-03-29T10:39:12.123 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction check succeeded. 2024-03-29T10:39:12.123 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction test 2024-03-29T10:39:12.225 INFO:teuthology.orchestra.run.smithi196.stdout:(8/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 51 MB/s | 16 MB 00:00 2024-03-29T10:39:12.250 INFO:teuthology.orchestra.run.smithi196.stdout:(9/12): libgfortran-11.4.1-3.el9.x86_64.rpm 2.6 MB/s | 803 kB 00:00 2024-03-29T10:39:12.299 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction test succeeded. 2024-03-29T10:39:12.299 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction 2024-03-29T10:39:12.358 INFO:teuthology.orchestra.run.smithi196.stdout:(10/12): libquadmath-11.4.1-3.el9.x86_64.rpm 1.4 MB/s | 195 kB 00:00 2024-03-29T10:39:12.408 INFO:teuthology.orchestra.run.smithi196.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 1.5 MB/s | 245 kB 00:00 2024-03-29T10:39:12.837 INFO:teuthology.orchestra.run.smithi088.stdout: Preparing : 1/1 2024-03-29T10:39:12.919 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-29T10:39:12.955 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-29T10:39:12.967 INFO:teuthology.orchestra.run.smithi196.stdout:(12/12): ceph-mgr-diskprediction-local-19.0.0-1 4.7 MB/s | 7.4 MB 00:01 2024-03-29T10:39:12.969 INFO:teuthology.orchestra.run.smithi196.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:39:12.969 INFO:teuthology.orchestra.run.smithi196.stdout:Total 22 MB/s | 37 MB 00:01 2024-03-29T10:39:13.001 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-29T10:39:13.009 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction check 2024-03-29T10:39:13.032 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-29T10:39:13.032 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction check succeeded. 2024-03-29T10:39:13.032 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction test 2024-03-29T10:39:13.133 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-29T10:39:13.257 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-29T10:39:13.352 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-29T10:39:13.386 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction test succeeded. 2024-03-29T10:39:13.387 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction 2024-03-29T10:39:13.395 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-29T10:39:13.423 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-29T10:39:13.463 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 10/12 2024-03-29T10:39:13.696 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 11/12 2024-03-29T10:39:13.730 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-29T10:39:14.084 INFO:teuthology.orchestra.run.smithi196.stdout: Preparing : 1/1 2024-03-29T10:39:14.165 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-29T10:39:14.205 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-29T10:39:14.297 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-29T10:39:14.341 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-29T10:39:14.519 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-29T10:39:14.521 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-29T10:39:14.521 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 1/12 2024-03-29T10:39:14.521 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 2/12 2024-03-29T10:39:14.521 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 3/12 2024-03-29T10:39:14.521 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-29T10:39:14.521 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-29T10:39:14.521 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-29T10:39:14.521 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-29T10:39:14.522 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-29T10:39:14.522 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-29T10:39:14.522 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-29T10:39:14.522 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-29T10:39:14.549 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-29T10:39:14.634 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-29T10:39:14.877 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-29T10:39:14.877 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-03-29T10:39:14.877 INFO:teuthology.orchestra.run.smithi088.stdout:Installed: 2024-03-29T10:39:14.878 INFO:teuthology.orchestra.run.smithi088.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-29T10:39:14.878 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:39:14.878 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:39:14.878 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:39:14.878 INFO:teuthology.orchestra.run.smithi088.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-29T10:39:14.878 INFO:teuthology.orchestra.run.smithi088.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-29T10:39:14.878 INFO:teuthology.orchestra.run.smithi088.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-29T10:39:14.878 INFO:teuthology.orchestra.run.smithi088.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-29T10:39:14.878 INFO:teuthology.orchestra.run.smithi088.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-29T10:39:14.878 INFO:teuthology.orchestra.run.smithi088.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-29T10:39:14.878 INFO:teuthology.orchestra.run.smithi088.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-29T10:39:14.878 INFO:teuthology.orchestra.run.smithi088.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-29T10:39:14.878 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-03-29T10:39:14.879 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2024-03-29T10:39:15.155 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-29T10:39:15.281 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-29T10:39:15.402 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-29T10:39:15.716 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:01:10 ago on Fri 29 Mar 2024 10:38:05 AM UTC. 2024-03-29T10:39:15.869 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2024-03-29T10:39:15.870 INFO:teuthology.orchestra.run.smithi088.stdout:============================================================================================ 2024-03-29T10:39:15.870 INFO:teuthology.orchestra.run.smithi088.stdout: Package Arch Version Repository Size 2024-03-29T10:39:15.870 INFO:teuthology.orchestra.run.smithi088.stdout:============================================================================================ 2024-03-29T10:39:15.870 INFO:teuthology.orchestra.run.smithi088.stdout:Installing: 2024-03-29T10:39:15.870 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 7.4 M 2024-03-29T10:39:15.871 INFO:teuthology.orchestra.run.smithi088.stdout:Installing dependencies: 2024-03-29T10:39:15.871 INFO:teuthology.orchestra.run.smithi088.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-29T10:39:15.871 INFO:teuthology.orchestra.run.smithi088.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-29T10:39:15.871 INFO:teuthology.orchestra.run.smithi088.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-29T10:39:15.871 INFO:teuthology.orchestra.run.smithi088.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-29T10:39:15.871 INFO:teuthology.orchestra.run.smithi088.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-29T10:39:15.871 INFO:teuthology.orchestra.run.smithi088.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-29T10:39:15.871 INFO:teuthology.orchestra.run.smithi088.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-29T10:39:15.871 INFO:teuthology.orchestra.run.smithi088.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-29T10:39:15.871 INFO:teuthology.orchestra.run.smithi088.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-29T10:39:15.871 INFO:teuthology.orchestra.run.smithi088.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-29T10:39:15.871 INFO:teuthology.orchestra.run.smithi088.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-29T10:39:15.872 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-03-29T10:39:15.872 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction Summary 2024-03-29T10:39:15.872 INFO:teuthology.orchestra.run.smithi088.stdout:============================================================================================ 2024-03-29T10:39:15.872 INFO:teuthology.orchestra.run.smithi088.stdout:Install 12 Packages 2024-03-29T10:39:15.872 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-03-29T10:39:15.872 INFO:teuthology.orchestra.run.smithi088.stdout:Total download size: 37 M 2024-03-29T10:39:15.873 INFO:teuthology.orchestra.run.smithi088.stdout:Installed size: 209 M 2024-03-29T10:39:15.873 INFO:teuthology.orchestra.run.smithi088.stdout:Downloading Packages: 2024-03-29T10:39:16.317 INFO:teuthology.orchestra.run.smithi088.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 221 kB/s | 32 kB 00:00 2024-03-29T10:39:16.350 INFO:teuthology.orchestra.run.smithi088.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 536 kB/s | 17 kB 00:00 2024-03-29T10:39:16.384 INFO:teuthology.orchestra.run.smithi088.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 1.0 MB/s | 34 kB 00:00 2024-03-29T10:39:16.451 INFO:teuthology.orchestra.run.smithi088.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 11 MB/s | 3.0 MB 00:00 2024-03-29T10:39:16.634 INFO:teuthology.orchestra.run.smithi088.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 19 MB/s | 4.7 MB 00:00 2024-03-29T10:39:16.641 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-29T10:39:16.701 INFO:teuthology.orchestra.run.smithi088.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 19 MB/s | 4.8 MB 00:00 2024-03-29T10:39:16.727 INFO:teuthology.orchestra.run.smithi088.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.3 MB/s | 217 kB 00:00 2024-03-29T10:39:17.027 INFO:teuthology.orchestra.run.smithi088.stdout:(8/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 48 MB/s | 16 MB 00:00 2024-03-29T10:39:17.669 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-29T10:39:17.699 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-29T10:39:17.927 INFO:teuthology.orchestra.run.smithi088.stdout:(9/12): ceph-mgr-diskprediction-local-19.0.0-14 4.2 MB/s | 7.4 MB 00:01 2024-03-29T10:39:17.952 INFO:teuthology.orchestra.run.smithi088.stdout:(10/12): libquadmath-11.4.1-3.el9.x86_64.rpm 211 kB/s | 195 kB 00:00 2024-03-29T10:39:18.186 INFO:teuthology.orchestra.run.smithi088.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 950 kB/s | 245 kB 00:00 2024-03-29T10:39:18.486 INFO:teuthology.orchestra.run.smithi088.stdout:(12/12): libgfortran-11.4.1-3.el9.x86_64.rpm 457 kB/s | 803 kB 00:01 2024-03-29T10:39:18.490 INFO:teuthology.orchestra.run.smithi088.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:39:18.490 INFO:teuthology.orchestra.run.smithi088.stdout:Total 14 MB/s | 37 MB 00:02 2024-03-29T10:39:18.534 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction check 2024-03-29T10:39:18.560 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction check succeeded. 2024-03-29T10:39:18.560 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction test 2024-03-29T10:39:18.918 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction test succeeded. 2024-03-29T10:39:18.919 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction 2024-03-29T10:39:19.611 INFO:teuthology.orchestra.run.smithi088.stdout: Preparing : 1/1 2024-03-29T10:39:19.692 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-29T10:39:19.723 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-29T10:39:19.842 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-29T10:39:19.885 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-29T10:39:20.054 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-29T10:39:20.075 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-29T10:39:20.153 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-29T10:39:20.258 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-29T10:39:20.258 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 1/12 2024-03-29T10:39:20.258 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-29T10:39:20.258 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-29T10:39:20.258 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-29T10:39:20.258 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-29T10:39:20.259 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-29T10:39:20.259 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-29T10:39:20.259 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-29T10:39:20.259 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-29T10:39:20.259 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-29T10:39:20.259 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-29T10:39:20.623 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-29T10:39:20.623 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-29T10:39:20.623 INFO:teuthology.orchestra.run.smithi196.stdout:Installed: 2024-03-29T10:39:20.623 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:39:20.623 INFO:teuthology.orchestra.run.smithi196.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-29T10:39:20.623 INFO:teuthology.orchestra.run.smithi196.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-29T10:39:20.623 INFO:teuthology.orchestra.run.smithi196.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-29T10:39:20.623 INFO:teuthology.orchestra.run.smithi196.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-29T10:39:20.623 INFO:teuthology.orchestra.run.smithi196.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-29T10:39:20.623 INFO:teuthology.orchestra.run.smithi196.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-29T10:39:20.623 INFO:teuthology.orchestra.run.smithi196.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-29T10:39:20.623 INFO:teuthology.orchestra.run.smithi196.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-29T10:39:20.624 INFO:teuthology.orchestra.run.smithi196.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-29T10:39:20.624 INFO:teuthology.orchestra.run.smithi196.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-29T10:39:20.624 INFO:teuthology.orchestra.run.smithi196.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-29T10:39:20.624 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-29T10:39:20.624 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-03-29T10:39:20.807 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-29T10:39:20.913 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-29T10:39:20.966 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install ceph-mgr-rook 2024-03-29T10:39:21.540 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:01:17 ago on Fri 29 Mar 2024 10:38:04 AM UTC. 2024-03-29T10:39:21.693 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-03-29T10:39:21.694 INFO:teuthology.orchestra.run.smithi196.stdout:======================================================================================== 2024-03-29T10:39:21.694 INFO:teuthology.orchestra.run.smithi196.stdout: Package Arch Version Repository Size 2024-03-29T10:39:21.694 INFO:teuthology.orchestra.run.smithi196.stdout:======================================================================================== 2024-03-29T10:39:21.694 INFO:teuthology.orchestra.run.smithi196.stdout:Installing: 2024-03-29T10:39:21.694 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mgr-rook noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 52 k 2024-03-29T10:39:21.694 INFO:teuthology.orchestra.run.smithi196.stdout:Installing dependencies: 2024-03-29T10:39:21.695 INFO:teuthology.orchestra.run.smithi196.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-29T10:39:21.695 INFO:teuthology.orchestra.run.smithi196.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-29T10:39:21.695 INFO:teuthology.orchestra.run.smithi196.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-29T10:39:21.695 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-29T10:39:21.695 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-29T10:39:21.695 INFO:teuthology.orchestra.run.smithi196.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-29T10:39:21.695 INFO:teuthology.orchestra.run.smithi196.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-29T10:39:21.695 INFO:teuthology.orchestra.run.smithi196.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-29T10:39:21.695 INFO:teuthology.orchestra.run.smithi196.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-29T10:39:21.696 INFO:teuthology.orchestra.run.smithi196.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-29T10:39:21.696 INFO:teuthology.orchestra.run.smithi196.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-29T10:39:21.696 INFO:teuthology.orchestra.run.smithi196.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-29T10:39:21.696 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-29T10:39:21.696 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction Summary 2024-03-29T10:39:21.696 INFO:teuthology.orchestra.run.smithi196.stdout:======================================================================================== 2024-03-29T10:39:21.696 INFO:teuthology.orchestra.run.smithi196.stdout:Install 13 Packages 2024-03-29T10:39:21.696 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-29T10:39:21.697 INFO:teuthology.orchestra.run.smithi196.stdout:Total download size: 2.0 M 2024-03-29T10:39:21.697 INFO:teuthology.orchestra.run.smithi196.stdout:Installed size: 26 M 2024-03-29T10:39:21.697 INFO:teuthology.orchestra.run.smithi196.stdout:Downloading Packages: 2024-03-29T10:39:21.916 INFO:teuthology.orchestra.run.smithi196.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 189 kB/s | 24 kB 00:00 2024-03-29T10:39:21.942 INFO:teuthology.orchestra.run.smithi196.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 117 kB/s | 18 kB 00:00 2024-03-29T10:39:21.992 INFO:teuthology.orchestra.run.smithi196.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.3 MB/s | 174 kB 00:00 2024-03-29T10:39:22.026 INFO:teuthology.orchestra.run.smithi196.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.5 MB/s | 132 kB 00:00 2024-03-29T10:39:22.051 INFO:teuthology.orchestra.run.smithi196.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 3.5 MB/s | 211 kB 00:00 2024-03-29T10:39:22.076 INFO:teuthology.orchestra.run.smithi196.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 860 kB/s | 43 kB 00:00 2024-03-29T10:39:22.102 INFO:teuthology.orchestra.run.smithi196.stdout:(7/13): ceph-mgr-rook-19.0.0-1439.g5dcb190d.el9 166 kB/s | 52 kB 00:00 2024-03-29T10:39:22.140 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-29T10:39:22.227 INFO:teuthology.orchestra.run.smithi196.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 99 kB/s | 15 kB 00:00 2024-03-29T10:39:22.252 INFO:teuthology.orchestra.run.smithi196.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 160 kB/s | 32 kB 00:00 2024-03-29T10:39:22.344 INFO:teuthology.orchestra.run.smithi196.stdout:(10/13): python3-google-auth-2.28.2-1.el9.noarc 837 kB/s | 203 kB 00:00 2024-03-29T10:39:22.369 INFO:teuthology.orchestra.run.smithi196.stdout:(11/13): python3-rsa-4.9-2.el9.noarch.rpm 505 kB/s | 59 kB 00:00 2024-03-29T10:39:22.419 INFO:teuthology.orchestra.run.smithi196.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.2 MB/s | 90 kB 00:00 2024-03-29T10:39:22.494 INFO:teuthology.orchestra.run.smithi196.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 3.8 MB/s | 1.0 MB 00:00 2024-03-29T10:39:22.496 INFO:teuthology.orchestra.run.smithi196.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:39:22.497 INFO:teuthology.orchestra.run.smithi196.stdout:Total 2.6 MB/s | 2.0 MB 00:00 2024-03-29T10:39:22.497 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction check 2024-03-29T10:39:22.510 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction check succeeded. 2024-03-29T10:39:22.510 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction test 2024-03-29T10:39:22.601 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction test succeeded. 2024-03-29T10:39:22.601 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction 2024-03-29T10:39:23.126 INFO:teuthology.orchestra.run.smithi196.stdout: Preparing : 1/1 2024-03-29T10:39:23.173 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-29T10:39:23.199 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-29T10:39:23.300 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-29T10:39:23.362 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-29T10:39:23.417 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-29T10:39:23.447 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-29T10:39:23.488 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-29T10:39:23.582 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-29T10:39:23.690 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-29T10:39:23.740 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-29T10:39:24.459 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-29T10:39:24.512 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-29T10:39:24.545 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-29T10:39:24.585 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-29T10:39:24.607 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-29T10:39:25.374 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-29T10:39:25.374 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 1/13 2024-03-29T10:39:25.374 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-29T10:39:25.374 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-29T10:39:25.374 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-29T10:39:25.374 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-29T10:39:25.374 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-29T10:39:25.374 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-29T10:39:25.374 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-29T10:39:25.374 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-29T10:39:25.374 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-29T10:39:25.374 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-29T10:39:25.375 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-29T10:39:25.719 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-29T10:39:25.719 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 1/12 2024-03-29T10:39:25.719 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-29T10:39:25.719 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-29T10:39:25.719 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-29T10:39:25.719 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-29T10:39:25.719 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-29T10:39:25.720 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-29T10:39:25.720 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-29T10:39:25.720 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-29T10:39:25.720 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-29T10:39:25.720 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-29T10:39:25.736 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-29T10:39:25.736 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-29T10:39:25.736 INFO:teuthology.orchestra.run.smithi196.stdout:Installed: 2024-03-29T10:39:25.736 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:39:25.737 INFO:teuthology.orchestra.run.smithi196.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-29T10:39:25.737 INFO:teuthology.orchestra.run.smithi196.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-29T10:39:25.737 INFO:teuthology.orchestra.run.smithi196.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-29T10:39:25.737 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-29T10:39:25.737 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-29T10:39:25.737 INFO:teuthology.orchestra.run.smithi196.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-29T10:39:25.737 INFO:teuthology.orchestra.run.smithi196.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-29T10:39:25.737 INFO:teuthology.orchestra.run.smithi196.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-29T10:39:25.737 INFO:teuthology.orchestra.run.smithi196.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-29T10:39:25.737 INFO:teuthology.orchestra.run.smithi196.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-29T10:39:25.737 INFO:teuthology.orchestra.run.smithi196.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-29T10:39:25.737 INFO:teuthology.orchestra.run.smithi196.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-29T10:39:25.737 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-29T10:39:25.737 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-03-29T10:39:25.993 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install ceph-mgr-cephadm 2024-03-29T10:39:26.177 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-29T10:39:26.177 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-03-29T10:39:26.177 INFO:teuthology.orchestra.run.smithi088.stdout:Installed: 2024-03-29T10:39:26.177 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:39:26.177 INFO:teuthology.orchestra.run.smithi088.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-29T10:39:26.177 INFO:teuthology.orchestra.run.smithi088.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-29T10:39:26.177 INFO:teuthology.orchestra.run.smithi088.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-29T10:39:26.177 INFO:teuthology.orchestra.run.smithi088.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-29T10:39:26.177 INFO:teuthology.orchestra.run.smithi088.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-29T10:39:26.178 INFO:teuthology.orchestra.run.smithi088.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-29T10:39:26.178 INFO:teuthology.orchestra.run.smithi088.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-29T10:39:26.178 INFO:teuthology.orchestra.run.smithi088.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-29T10:39:26.178 INFO:teuthology.orchestra.run.smithi088.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-29T10:39:26.178 INFO:teuthology.orchestra.run.smithi088.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-29T10:39:26.178 INFO:teuthology.orchestra.run.smithi088.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-29T10:39:26.178 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-03-29T10:39:26.178 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2024-03-29T10:39:26.548 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install ceph-mgr-rook 2024-03-29T10:39:26.561 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:01:22 ago on Fri 29 Mar 2024 10:38:04 AM UTC. 2024-03-29T10:39:26.713 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-03-29T10:39:26.714 INFO:teuthology.orchestra.run.smithi196.stdout:======================================================================================== 2024-03-29T10:39:26.714 INFO:teuthology.orchestra.run.smithi196.stdout: Package Arch Version Repository Size 2024-03-29T10:39:26.714 INFO:teuthology.orchestra.run.smithi196.stdout:======================================================================================== 2024-03-29T10:39:26.714 INFO:teuthology.orchestra.run.smithi196.stdout:Installing: 2024-03-29T10:39:26.714 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 138 k 2024-03-29T10:39:26.714 INFO:teuthology.orchestra.run.smithi196.stdout:Installing dependencies: 2024-03-29T10:39:26.714 INFO:teuthology.orchestra.run.smithi196.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-29T10:39:26.714 INFO:teuthology.orchestra.run.smithi196.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-29T10:39:26.714 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-29T10:39:26.714 INFO:teuthology.orchestra.run.smithi196.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-29T10:39:26.714 INFO:teuthology.orchestra.run.smithi196.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-29T10:39:26.714 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-29T10:39:26.715 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction Summary 2024-03-29T10:39:26.715 INFO:teuthology.orchestra.run.smithi196.stdout:======================================================================================== 2024-03-29T10:39:26.715 INFO:teuthology.orchestra.run.smithi196.stdout:Install 6 Packages 2024-03-29T10:39:26.715 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-29T10:39:26.715 INFO:teuthology.orchestra.run.smithi196.stdout:Total download size: 6.8 M 2024-03-29T10:39:26.715 INFO:teuthology.orchestra.run.smithi196.stdout:Installed size: 33 M 2024-03-29T10:39:26.716 INFO:teuthology.orchestra.run.smithi196.stdout:Downloading Packages: 2024-03-29T10:39:27.117 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:01:22 ago on Fri 29 Mar 2024 10:38:05 AM UTC. 2024-03-29T10:39:27.214 INFO:teuthology.orchestra.run.smithi196.stdout:(1/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 1.4 MB/s | 248 kB 00:00 2024-03-29T10:39:27.268 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2024-03-29T10:39:27.269 INFO:teuthology.orchestra.run.smithi088.stdout:======================================================================================== 2024-03-29T10:39:27.269 INFO:teuthology.orchestra.run.smithi088.stdout: Package Arch Version Repository Size 2024-03-29T10:39:27.270 INFO:teuthology.orchestra.run.smithi088.stdout:======================================================================================== 2024-03-29T10:39:27.270 INFO:teuthology.orchestra.run.smithi088.stdout:Installing: 2024-03-29T10:39:27.270 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mgr-rook noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 52 k 2024-03-29T10:39:27.270 INFO:teuthology.orchestra.run.smithi088.stdout:Installing dependencies: 2024-03-29T10:39:27.270 INFO:teuthology.orchestra.run.smithi088.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-29T10:39:27.270 INFO:teuthology.orchestra.run.smithi088.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-29T10:39:27.270 INFO:teuthology.orchestra.run.smithi088.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-29T10:39:27.270 INFO:teuthology.orchestra.run.smithi088.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-29T10:39:27.270 INFO:teuthology.orchestra.run.smithi088.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-29T10:39:27.270 INFO:teuthology.orchestra.run.smithi088.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-29T10:39:27.270 INFO:teuthology.orchestra.run.smithi088.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-29T10:39:27.270 INFO:teuthology.orchestra.run.smithi088.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-29T10:39:27.270 INFO:teuthology.orchestra.run.smithi088.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-29T10:39:27.271 INFO:teuthology.orchestra.run.smithi088.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-29T10:39:27.271 INFO:teuthology.orchestra.run.smithi088.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-29T10:39:27.271 INFO:teuthology.orchestra.run.smithi088.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-29T10:39:27.271 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-03-29T10:39:27.271 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction Summary 2024-03-29T10:39:27.271 INFO:teuthology.orchestra.run.smithi088.stdout:======================================================================================== 2024-03-29T10:39:27.271 INFO:teuthology.orchestra.run.smithi088.stdout:Install 13 Packages 2024-03-29T10:39:27.271 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-03-29T10:39:27.272 INFO:teuthology.orchestra.run.smithi088.stdout:Total download size: 2.0 M 2024-03-29T10:39:27.272 INFO:teuthology.orchestra.run.smithi088.stdout:Installed size: 26 M 2024-03-29T10:39:27.272 INFO:teuthology.orchestra.run.smithi088.stdout:Downloading Packages: 2024-03-29T10:39:27.422 INFO:teuthology.orchestra.run.smithi196.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 15 MB/s | 5.8 MB 00:00 2024-03-29T10:39:27.491 INFO:teuthology.orchestra.run.smithi088.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 168 kB/s | 24 kB 00:00 2024-03-29T10:39:27.514 INFO:teuthology.orchestra.run.smithi196.stdout:(3/6): ceph-mgr-cephadm-19.0.0-1439.g5dcb190d.e 292 kB/s | 138 kB 00:00 2024-03-29T10:39:27.516 INFO:teuthology.orchestra.run.smithi088.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 106 kB/s | 18 kB 00:00 2024-03-29T10:39:27.542 INFO:teuthology.orchestra.run.smithi088.stdout:(3/13): ceph-mgr-rook-19.0.0-1439.g5dcb190d.el9 267 kB/s | 52 kB 00:00 2024-03-29T10:39:27.575 INFO:teuthology.orchestra.run.smithi088.stdout:(4/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.0 MB/s | 174 kB 00:00 2024-03-29T10:39:27.601 INFO:teuthology.orchestra.run.smithi088.stdout:(5/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.5 MB/s | 132 kB 00:00 2024-03-29T10:39:27.626 INFO:teuthology.orchestra.run.smithi088.stdout:(6/13): python3-pyasn1-modules-0.4.8-6.el9.noar 2.5 MB/s | 211 kB 00:00 2024-03-29T10:39:27.651 INFO:teuthology.orchestra.run.smithi088.stdout:(7/13): python3-requests-oauthlib-1.3.0-12.el9. 569 kB/s | 43 kB 00:00 2024-03-29T10:39:27.702 INFO:teuthology.orchestra.run.smithi088.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 299 kB/s | 15 kB 00:00 2024-03-29T10:39:27.727 INFO:teuthology.orchestra.run.smithi088.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 256 kB/s | 32 kB 00:00 2024-03-29T10:39:27.769 INFO:teuthology.orchestra.run.smithi088.stdout:(10/13): python3-google-auth-2.28.2-1.el9.noarc 1.7 MB/s | 203 kB 00:00 2024-03-29T10:39:27.794 INFO:teuthology.orchestra.run.smithi088.stdout:(11/13): python3-rsa-4.9-2.el9.noarch.rpm 884 kB/s | 59 kB 00:00 2024-03-29T10:39:27.820 INFO:teuthology.orchestra.run.smithi088.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.7 MB/s | 90 kB 00:00 2024-03-29T10:39:27.870 INFO:teuthology.orchestra.run.smithi088.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 6.1 MB/s | 1.0 MB 00:00 2024-03-29T10:39:27.871 INFO:teuthology.orchestra.run.smithi088.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:39:27.872 INFO:teuthology.orchestra.run.smithi088.stdout:Total 3.4 MB/s | 2.0 MB 00:00 2024-03-29T10:39:27.872 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction check 2024-03-29T10:39:27.885 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction check succeeded. 2024-03-29T10:39:27.885 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction test 2024-03-29T10:39:27.898 INFO:teuthology.orchestra.run.smithi196.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 122 kB/s | 58 kB 00:00 2024-03-29T10:39:27.956 INFO:teuthology.orchestra.run.smithi196.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 122 kB/s | 51 kB 00:00 2024-03-29T10:39:27.975 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction test succeeded. 2024-03-29T10:39:27.976 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction 2024-03-29T10:39:27.981 INFO:teuthology.orchestra.run.smithi196.stdout:(6/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 715 kB/s | 548 kB 00:00 2024-03-29T10:39:27.985 INFO:teuthology.orchestra.run.smithi196.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:39:27.985 INFO:teuthology.orchestra.run.smithi196.stdout:Total 5.3 MB/s | 6.8 MB 00:01 2024-03-29T10:39:27.999 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction check 2024-03-29T10:39:28.009 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction check succeeded. 2024-03-29T10:39:28.009 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction test 2024-03-29T10:39:28.097 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction test succeeded. 2024-03-29T10:39:28.099 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction 2024-03-29T10:39:28.397 INFO:teuthology.orchestra.run.smithi196.stdout: Preparing : 1/1 2024-03-29T10:39:28.497 INFO:teuthology.orchestra.run.smithi088.stdout: Preparing : 1/1 2024-03-29T10:39:28.588 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-29T10:39:28.643 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-29T10:39:28.647 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-29T10:39:28.714 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-29T10:39:28.767 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-29T10:39:28.805 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-29T10:39:28.846 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-29T10:39:28.938 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-29T10:39:29.047 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-29T10:39:29.097 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-29T10:39:29.160 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-29T10:39:29.246 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-29T10:39:29.307 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-29T10:39:29.331 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-29T10:39:29.811 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-29T10:39:29.870 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-29T10:39:29.903 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-29T10:39:29.952 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-29T10:39:29.973 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-29T10:39:30.121 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-29T10:39:30.121 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/6 2024-03-29T10:39:30.121 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-29T10:39:30.121 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-29T10:39:30.121 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-29T10:39:30.121 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-29T10:39:30.438 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-29T10:39:30.438 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-29T10:39:30.438 INFO:teuthology.orchestra.run.smithi196.stdout:Installed: 2024-03-29T10:39:30.438 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:39:30.439 INFO:teuthology.orchestra.run.smithi196.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-29T10:39:30.439 INFO:teuthology.orchestra.run.smithi196.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-29T10:39:30.439 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-29T10:39:30.439 INFO:teuthology.orchestra.run.smithi196.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-29T10:39:30.439 INFO:teuthology.orchestra.run.smithi196.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-29T10:39:30.439 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-29T10:39:30.439 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-03-29T10:39:30.726 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install ceph-fuse 2024-03-29T10:39:30.763 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-29T10:39:30.763 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 1/13 2024-03-29T10:39:30.763 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-29T10:39:30.763 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-29T10:39:30.763 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-29T10:39:30.763 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-29T10:39:30.763 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-29T10:39:30.763 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-29T10:39:30.763 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-29T10:39:30.763 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-29T10:39:30.763 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-29T10:39:30.764 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-29T10:39:30.764 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-29T10:39:31.122 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-29T10:39:31.123 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-03-29T10:39:31.123 INFO:teuthology.orchestra.run.smithi088.stdout:Installed: 2024-03-29T10:39:31.123 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:39:31.123 INFO:teuthology.orchestra.run.smithi088.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-29T10:39:31.123 INFO:teuthology.orchestra.run.smithi088.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-29T10:39:31.123 INFO:teuthology.orchestra.run.smithi088.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-29T10:39:31.123 INFO:teuthology.orchestra.run.smithi088.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-29T10:39:31.123 INFO:teuthology.orchestra.run.smithi088.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-29T10:39:31.123 INFO:teuthology.orchestra.run.smithi088.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-29T10:39:31.123 INFO:teuthology.orchestra.run.smithi088.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-29T10:39:31.123 INFO:teuthology.orchestra.run.smithi088.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-29T10:39:31.123 INFO:teuthology.orchestra.run.smithi088.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-29T10:39:31.123 INFO:teuthology.orchestra.run.smithi088.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-29T10:39:31.124 INFO:teuthology.orchestra.run.smithi088.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-29T10:39:31.124 INFO:teuthology.orchestra.run.smithi088.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-29T10:39:31.124 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-03-29T10:39:31.124 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2024-03-29T10:39:31.289 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:01:27 ago on Fri 29 Mar 2024 10:38:04 AM UTC. 2024-03-29T10:39:31.415 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install ceph-mgr-cephadm 2024-03-29T10:39:31.442 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-03-29T10:39:31.443 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-29T10:39:31.443 INFO:teuthology.orchestra.run.smithi196.stdout: Package Arch Version Repository Size 2024-03-29T10:39:31.443 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-29T10:39:31.443 INFO:teuthology.orchestra.run.smithi196.stdout:Installing: 2024-03-29T10:39:31.443 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 885 k 2024-03-29T10:39:31.443 INFO:teuthology.orchestra.run.smithi196.stdout:Installing dependencies: 2024-03-29T10:39:31.444 INFO:teuthology.orchestra.run.smithi196.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-29T10:39:31.444 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-29T10:39:31.444 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction Summary 2024-03-29T10:39:31.444 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-29T10:39:31.444 INFO:teuthology.orchestra.run.smithi196.stdout:Install 2 Packages 2024-03-29T10:39:31.444 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-29T10:39:31.444 INFO:teuthology.orchestra.run.smithi196.stdout:Total download size: 964 k 2024-03-29T10:39:31.444 INFO:teuthology.orchestra.run.smithi196.stdout:Installed size: 2.8 M 2024-03-29T10:39:31.444 INFO:teuthology.orchestra.run.smithi196.stdout:Downloading Packages: 2024-03-29T10:39:31.760 INFO:teuthology.orchestra.run.smithi196.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 247 kB/s | 79 kB 00:00 2024-03-29T10:39:31.984 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:01:26 ago on Fri 29 Mar 2024 10:38:05 AM UTC. 2024-03-29T10:39:32.134 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2024-03-29T10:39:32.135 INFO:teuthology.orchestra.run.smithi088.stdout:======================================================================================== 2024-03-29T10:39:32.135 INFO:teuthology.orchestra.run.smithi088.stdout: Package Arch Version Repository Size 2024-03-29T10:39:32.135 INFO:teuthology.orchestra.run.smithi088.stdout:======================================================================================== 2024-03-29T10:39:32.135 INFO:teuthology.orchestra.run.smithi088.stdout:Installing: 2024-03-29T10:39:32.135 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 138 k 2024-03-29T10:39:32.135 INFO:teuthology.orchestra.run.smithi088.stdout:Installing dependencies: 2024-03-29T10:39:32.135 INFO:teuthology.orchestra.run.smithi088.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-29T10:39:32.135 INFO:teuthology.orchestra.run.smithi088.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-29T10:39:32.135 INFO:teuthology.orchestra.run.smithi088.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-29T10:39:32.135 INFO:teuthology.orchestra.run.smithi088.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-29T10:39:32.136 INFO:teuthology.orchestra.run.smithi088.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-29T10:39:32.136 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-03-29T10:39:32.136 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction Summary 2024-03-29T10:39:32.136 INFO:teuthology.orchestra.run.smithi088.stdout:======================================================================================== 2024-03-29T10:39:32.136 INFO:teuthology.orchestra.run.smithi088.stdout:Install 6 Packages 2024-03-29T10:39:32.136 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-03-29T10:39:32.136 INFO:teuthology.orchestra.run.smithi088.stdout:Total download size: 6.8 M 2024-03-29T10:39:32.137 INFO:teuthology.orchestra.run.smithi088.stdout:Installed size: 33 M 2024-03-29T10:39:32.137 INFO:teuthology.orchestra.run.smithi088.stdout:Downloading Packages: 2024-03-29T10:39:32.235 INFO:teuthology.orchestra.run.smithi196.stdout:(2/2): ceph-fuse-19.0.0-1439.g5dcb190d.el9.x86_ 1.1 MB/s | 885 kB 00:00 2024-03-29T10:39:32.236 INFO:teuthology.orchestra.run.smithi196.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:39:32.237 INFO:teuthology.orchestra.run.smithi196.stdout:Total 1.2 MB/s | 964 kB 00:00 2024-03-29T10:39:32.237 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction check 2024-03-29T10:39:32.257 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction check succeeded. 2024-03-29T10:39:32.257 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction test 2024-03-29T10:39:32.330 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction test succeeded. 2024-03-29T10:39:32.331 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction 2024-03-29T10:39:32.511 INFO:teuthology.orchestra.run.smithi196.stdout: Preparing : 1/1 2024-03-29T10:39:32.644 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-29T10:39:32.719 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-29T10:39:32.729 INFO:teuthology.orchestra.run.smithi088.stdout:(1/6): python3-babel-2.9.1-2.el9.noarch.rpm 12 MB/s | 5.8 MB 00:00 2024-03-29T10:39:32.754 INFO:teuthology.orchestra.run.smithi088.stdout:(2/6): ceph-mgr-cephadm-19.0.0-1439.g5dcb190d.e 275 kB/s | 138 kB 00:00 2024-03-29T10:39:32.780 INFO:teuthology.orchestra.run.smithi088.stdout:(3/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 471 kB/s | 248 kB 00:00 2024-03-29T10:39:32.971 INFO:teuthology.orchestra.run.smithi088.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 266 kB/s | 58 kB 00:00 2024-03-29T10:39:32.996 INFO:teuthology.orchestra.run.smithi088.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 235 kB/s | 51 kB 00:00 2024-03-29T10:39:33.213 INFO:teuthology.orchestra.run.smithi088.stdout:(6/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 1.1 MB/s | 548 kB 00:00 2024-03-29T10:39:33.216 INFO:teuthology.orchestra.run.smithi088.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:39:33.216 INFO:teuthology.orchestra.run.smithi088.stdout:Total 6.3 MB/s | 6.8 MB 00:01 2024-03-29T10:39:33.230 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction check 2024-03-29T10:39:33.240 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction check succeeded. 2024-03-29T10:39:33.240 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction test 2024-03-29T10:39:33.329 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction test succeeded. 2024-03-29T10:39:33.330 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction 2024-03-29T10:39:33.443 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-29T10:39:33.443 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-29T10:39:33.616 INFO:teuthology.orchestra.run.smithi088.stdout: Preparing : 1/1 2024-03-29T10:39:33.733 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-29T10:39:33.733 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-29T10:39:33.733 INFO:teuthology.orchestra.run.smithi196.stdout:Installed: 2024-03-29T10:39:33.733 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-29T10:39:33.734 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-29T10:39:33.734 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-03-29T10:39:33.770 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-29T10:39:33.815 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-29T10:39:34.048 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install ceph-volume 2024-03-29T10:39:34.333 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-29T10:39:34.410 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-29T10:39:34.473 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-29T10:39:34.495 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-29T10:39:34.626 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:01:30 ago on Fri 29 Mar 2024 10:38:04 AM UTC. 2024-03-29T10:39:34.784 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-03-29T10:39:34.785 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-29T10:39:34.785 INFO:teuthology.orchestra.run.smithi196.stdout: Package Arch Version Repository Size 2024-03-29T10:39:34.785 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-29T10:39:34.785 INFO:teuthology.orchestra.run.smithi196.stdout:Installing: 2024-03-29T10:39:34.785 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-volume noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 264 k 2024-03-29T10:39:34.785 INFO:teuthology.orchestra.run.smithi196.stdout:Installing dependencies: 2024-03-29T10:39:34.785 INFO:teuthology.orchestra.run.smithi196.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-29T10:39:34.785 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-29T10:39:34.785 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction Summary 2024-03-29T10:39:34.785 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-29T10:39:34.786 INFO:teuthology.orchestra.run.smithi196.stdout:Install 2 Packages 2024-03-29T10:39:34.786 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-29T10:39:34.786 INFO:teuthology.orchestra.run.smithi196.stdout:Total download size: 332 k 2024-03-29T10:39:34.786 INFO:teuthology.orchestra.run.smithi196.stdout:Installed size: 1.5 M 2024-03-29T10:39:34.787 INFO:teuthology.orchestra.run.smithi196.stdout:Downloading Packages: 2024-03-29T10:39:35.015 INFO:teuthology.orchestra.run.smithi196.stdout:(1/2): ceph-volume-19.0.0-1439.g5dcb190d.el9.no 1.1 MB/s | 264 kB 00:00 2024-03-29T10:39:35.065 INFO:teuthology.orchestra.run.smithi196.stdout:(2/2): python3-packaging-20.9-5.el9.noarch.rpm 246 kB/s | 69 kB 00:00 2024-03-29T10:39:35.066 INFO:teuthology.orchestra.run.smithi196.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:39:35.067 INFO:teuthology.orchestra.run.smithi196.stdout:Total 1.2 MB/s | 332 kB 00:00 2024-03-29T10:39:35.067 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction check 2024-03-29T10:39:35.075 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction check succeeded. 2024-03-29T10:39:35.075 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction test 2024-03-29T10:39:35.113 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction test succeeded. 2024-03-29T10:39:35.113 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction 2024-03-29T10:39:35.270 INFO:teuthology.orchestra.run.smithi196.stdout: Preparing : 1/1 2024-03-29T10:39:35.280 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-29T10:39:35.280 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/6 2024-03-29T10:39:35.280 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-29T10:39:35.280 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-29T10:39:35.280 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-29T10:39:35.280 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-29T10:39:35.447 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-29T10:39:35.485 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-29T10:39:35.502 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-29T10:39:35.502 INFO:teuthology.orchestra.run.smithi196.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:39:35.502 INFO:teuthology.orchestra.run.smithi196.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-29T10:39:35.502 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-29T10:39:35.569 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-29T10:39:35.569 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-03-29T10:39:35.570 INFO:teuthology.orchestra.run.smithi088.stdout:Installed: 2024-03-29T10:39:35.570 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:39:35.570 INFO:teuthology.orchestra.run.smithi088.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-29T10:39:35.570 INFO:teuthology.orchestra.run.smithi088.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-29T10:39:35.570 INFO:teuthology.orchestra.run.smithi088.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-29T10:39:35.570 INFO:teuthology.orchestra.run.smithi088.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-29T10:39:35.570 INFO:teuthology.orchestra.run.smithi088.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-29T10:39:35.570 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-03-29T10:39:35.570 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2024-03-29T10:39:35.841 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install ceph-fuse 2024-03-29T10:39:36.255 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 1/2 2024-03-29T10:39:36.398 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:01:31 ago on Fri 29 Mar 2024 10:38:05 AM UTC. 2024-03-29T10:39:36.548 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2024-03-29T10:39:36.548 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-03-29T10:39:36.548 INFO:teuthology.orchestra.run.smithi088.stdout: Package Arch Version Repository Size 2024-03-29T10:39:36.548 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-03-29T10:39:36.549 INFO:teuthology.orchestra.run.smithi088.stdout:Installing: 2024-03-29T10:39:36.549 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 885 k 2024-03-29T10:39:36.549 INFO:teuthology.orchestra.run.smithi088.stdout:Installing dependencies: 2024-03-29T10:39:36.549 INFO:teuthology.orchestra.run.smithi088.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-29T10:39:36.549 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-03-29T10:39:36.549 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction Summary 2024-03-29T10:39:36.549 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-03-29T10:39:36.549 INFO:teuthology.orchestra.run.smithi088.stdout:Install 2 Packages 2024-03-29T10:39:36.549 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-03-29T10:39:36.550 INFO:teuthology.orchestra.run.smithi088.stdout:Total download size: 964 k 2024-03-29T10:39:36.550 INFO:teuthology.orchestra.run.smithi088.stdout:Installed size: 2.8 M 2024-03-29T10:39:36.550 INFO:teuthology.orchestra.run.smithi088.stdout:Downloading Packages: 2024-03-29T10:39:36.560 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-29T10:39:36.560 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-29T10:39:36.560 INFO:teuthology.orchestra.run.smithi196.stdout:Installed: 2024-03-29T10:39:36.560 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:39:36.560 INFO:teuthology.orchestra.run.smithi196.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-29T10:39:36.561 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-29T10:39:36.561 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-03-29T10:39:36.700 INFO:teuthology.orchestra.run.smithi088.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 526 kB/s | 79 kB 00:00 2024-03-29T10:39:36.866 INFO:teuthology.orchestra.run.smithi088.stdout:(2/2): ceph-fuse-19.0.0-1439.g5dcb190d.el9.x86_ 2.7 MB/s | 885 kB 00:00 2024-03-29T10:39:36.867 INFO:teuthology.orchestra.run.smithi088.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:39:36.868 INFO:teuthology.orchestra.run.smithi088.stdout:Total 3.0 MB/s | 964 kB 00:00 2024-03-29T10:39:36.868 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction check 2024-03-29T10:39:36.888 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction check succeeded. 2024-03-29T10:39:36.888 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction test 2024-03-29T10:39:36.942 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install librados-devel 2024-03-29T10:39:36.961 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction test succeeded. 2024-03-29T10:39:36.961 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction 2024-03-29T10:39:37.140 INFO:teuthology.orchestra.run.smithi088.stdout: Preparing : 1/1 2024-03-29T10:39:37.284 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-29T10:39:37.349 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-29T10:39:37.523 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:01:33 ago on Fri 29 Mar 2024 10:38:04 AM UTC. 2024-03-29T10:39:37.677 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-03-29T10:39:37.677 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-29T10:39:37.677 INFO:teuthology.orchestra.run.smithi196.stdout: Package Arch Version Repo Size 2024-03-29T10:39:37.677 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-29T10:39:37.678 INFO:teuthology.orchestra.run.smithi196.stdout:Installing: 2024-03-29T10:39:37.678 INFO:teuthology.orchestra.run.smithi196.stdout: librados-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 127 k 2024-03-29T10:39:37.678 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-29T10:39:37.678 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction Summary 2024-03-29T10:39:37.678 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-29T10:39:37.678 INFO:teuthology.orchestra.run.smithi196.stdout:Install 1 Package 2024-03-29T10:39:37.678 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-29T10:39:37.679 INFO:teuthology.orchestra.run.smithi196.stdout:Total download size: 127 k 2024-03-29T10:39:37.679 INFO:teuthology.orchestra.run.smithi196.stdout:Installed size: 456 k 2024-03-29T10:39:37.679 INFO:teuthology.orchestra.run.smithi196.stdout:Downloading Packages: 2024-03-29T10:39:38.030 INFO:teuthology.orchestra.run.smithi196.stdout:librados-devel-19.0.0-1439.g5dcb190d.el9.x86_64 361 kB/s | 127 kB 00:00 2024-03-29T10:39:38.030 INFO:teuthology.orchestra.run.smithi196.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:39:38.031 INFO:teuthology.orchestra.run.smithi196.stdout:Total 359 kB/s | 127 kB 00:00 2024-03-29T10:39:38.031 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction check 2024-03-29T10:39:38.036 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction check succeeded. 2024-03-29T10:39:38.036 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction test 2024-03-29T10:39:38.040 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-29T10:39:38.040 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-29T10:39:38.091 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction test succeeded. 2024-03-29T10:39:38.092 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction 2024-03-29T10:39:38.242 INFO:teuthology.orchestra.run.smithi196.stdout: Preparing : 1/1 2024-03-29T10:39:38.321 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-29T10:39:38.321 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-03-29T10:39:38.321 INFO:teuthology.orchestra.run.smithi088.stdout:Installed: 2024-03-29T10:39:38.321 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-29T10:39:38.321 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-03-29T10:39:38.321 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2024-03-29T10:39:38.338 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:39:38.627 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install ceph-volume 2024-03-29T10:39:38.752 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:39:39.039 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:39:39.039 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-29T10:39:39.040 INFO:teuthology.orchestra.run.smithi196.stdout:Installed: 2024-03-29T10:39:39.040 INFO:teuthology.orchestra.run.smithi196.stdout: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:39:39.040 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-29T10:39:39.040 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-03-29T10:39:39.206 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:01:34 ago on Fri 29 Mar 2024 10:38:05 AM UTC. 2024-03-29T10:39:39.273 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install libcephfs2 2024-03-29T10:39:39.357 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2024-03-29T10:39:39.358 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-03-29T10:39:39.358 INFO:teuthology.orchestra.run.smithi088.stdout: Package Arch Version Repository Size 2024-03-29T10:39:39.359 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-03-29T10:39:39.359 INFO:teuthology.orchestra.run.smithi088.stdout:Installing: 2024-03-29T10:39:39.359 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-volume noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 264 k 2024-03-29T10:39:39.359 INFO:teuthology.orchestra.run.smithi088.stdout:Installing dependencies: 2024-03-29T10:39:39.359 INFO:teuthology.orchestra.run.smithi088.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-29T10:39:39.359 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-03-29T10:39:39.359 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction Summary 2024-03-29T10:39:39.359 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-03-29T10:39:39.359 INFO:teuthology.orchestra.run.smithi088.stdout:Install 2 Packages 2024-03-29T10:39:39.359 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-03-29T10:39:39.359 INFO:teuthology.orchestra.run.smithi088.stdout:Total download size: 332 k 2024-03-29T10:39:39.359 INFO:teuthology.orchestra.run.smithi088.stdout:Installed size: 1.5 M 2024-03-29T10:39:39.359 INFO:teuthology.orchestra.run.smithi088.stdout:Downloading Packages: 2024-03-29T10:39:39.479 INFO:teuthology.orchestra.run.smithi088.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 573 kB/s | 69 kB 00:00 2024-03-29T10:39:39.605 INFO:teuthology.orchestra.run.smithi088.stdout:(2/2): ceph-volume-19.0.0-1439.g5dcb190d.el9.no 1.1 MB/s | 264 kB 00:00 2024-03-29T10:39:39.605 INFO:teuthology.orchestra.run.smithi088.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:39:39.606 INFO:teuthology.orchestra.run.smithi088.stdout:Total 1.3 MB/s | 332 kB 00:00 2024-03-29T10:39:39.606 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction check 2024-03-29T10:39:39.614 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction check succeeded. 2024-03-29T10:39:39.614 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction test 2024-03-29T10:39:39.651 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction test succeeded. 2024-03-29T10:39:39.652 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction 2024-03-29T10:39:39.820 INFO:teuthology.orchestra.run.smithi088.stdout: Preparing : 1/1 2024-03-29T10:39:39.852 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:01:35 ago on Fri 29 Mar 2024 10:38:04 AM UTC. 2024-03-29T10:39:39.954 INFO:teuthology.orchestra.run.smithi196.stdout:Package libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:39:40.005 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-29T10:39:40.010 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-03-29T10:39:40.011 INFO:teuthology.orchestra.run.smithi196.stdout:Nothing to do. 2024-03-29T10:39:40.011 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-03-29T10:39:40.040 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-29T10:39:40.057 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-29T10:39:40.057 INFO:teuthology.orchestra.run.smithi088.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:39:40.057 INFO:teuthology.orchestra.run.smithi088.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-29T10:39:40.057 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-03-29T10:39:40.081 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install libcephfs-devel 2024-03-29T10:39:40.651 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:01:36 ago on Fri 29 Mar 2024 10:38:04 AM UTC. 2024-03-29T10:39:40.805 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-03-29T10:39:40.805 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-29T10:39:40.805 INFO:teuthology.orchestra.run.smithi196.stdout: Package Arch Version Repo Size 2024-03-29T10:39:40.805 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-29T10:39:40.806 INFO:teuthology.orchestra.run.smithi196.stdout:Installing: 2024-03-29T10:39:40.806 INFO:teuthology.orchestra.run.smithi196.stdout: libcephfs-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 31 k 2024-03-29T10:39:40.806 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-29T10:39:40.806 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction Summary 2024-03-29T10:39:40.806 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-29T10:39:40.806 INFO:teuthology.orchestra.run.smithi196.stdout:Install 1 Package 2024-03-29T10:39:40.806 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-29T10:39:40.806 INFO:teuthology.orchestra.run.smithi196.stdout:Total download size: 31 k 2024-03-29T10:39:40.806 INFO:teuthology.orchestra.run.smithi196.stdout:Installed size: 140 k 2024-03-29T10:39:40.806 INFO:teuthology.orchestra.run.smithi196.stdout:Downloading Packages: 2024-03-29T10:39:40.818 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 1/2 2024-03-29T10:39:41.050 INFO:teuthology.orchestra.run.smithi196.stdout:libcephfs-devel-19.0.0-1439.g5dcb190d.el9.x86_6 128 kB/s | 31 kB 00:00 2024-03-29T10:39:41.051 INFO:teuthology.orchestra.run.smithi196.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:39:41.051 INFO:teuthology.orchestra.run.smithi196.stdout:Total 127 kB/s | 31 kB 00:00 2024-03-29T10:39:41.051 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction check 2024-03-29T10:39:41.054 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction check succeeded. 2024-03-29T10:39:41.054 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction test 2024-03-29T10:39:41.093 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction test succeeded. 2024-03-29T10:39:41.093 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction 2024-03-29T10:39:41.172 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-29T10:39:41.172 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-03-29T10:39:41.172 INFO:teuthology.orchestra.run.smithi088.stdout:Installed: 2024-03-29T10:39:41.172 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:39:41.172 INFO:teuthology.orchestra.run.smithi088.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-29T10:39:41.172 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-03-29T10:39:41.172 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2024-03-29T10:39:41.220 INFO:teuthology.orchestra.run.smithi196.stdout: Preparing : 1/1 2024-03-29T10:39:41.323 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:39:41.526 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install librados-devel 2024-03-29T10:39:41.710 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:39:41.976 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:39:41.976 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-29T10:39:41.976 INFO:teuthology.orchestra.run.smithi196.stdout:Installed: 2024-03-29T10:39:41.976 INFO:teuthology.orchestra.run.smithi196.stdout: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:39:41.976 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-29T10:39:41.976 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-03-29T10:39:42.106 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:01:37 ago on Fri 29 Mar 2024 10:38:05 AM UTC. 2024-03-29T10:39:42.152 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install librados2 2024-03-29T10:39:42.257 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2024-03-29T10:39:42.258 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-03-29T10:39:42.258 INFO:teuthology.orchestra.run.smithi088.stdout: Package Arch Version Repo Size 2024-03-29T10:39:42.258 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-03-29T10:39:42.259 INFO:teuthology.orchestra.run.smithi088.stdout:Installing: 2024-03-29T10:39:42.259 INFO:teuthology.orchestra.run.smithi088.stdout: librados-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 127 k 2024-03-29T10:39:42.259 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-03-29T10:39:42.259 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction Summary 2024-03-29T10:39:42.259 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-03-29T10:39:42.259 INFO:teuthology.orchestra.run.smithi088.stdout:Install 1 Package 2024-03-29T10:39:42.259 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-03-29T10:39:42.259 INFO:teuthology.orchestra.run.smithi088.stdout:Total download size: 127 k 2024-03-29T10:39:42.259 INFO:teuthology.orchestra.run.smithi088.stdout:Installed size: 456 k 2024-03-29T10:39:42.260 INFO:teuthology.orchestra.run.smithi088.stdout:Downloading Packages: 2024-03-29T10:39:42.460 INFO:teuthology.orchestra.run.smithi088.stdout:librados-devel-19.0.0-1439.g5dcb190d.el9.x86_64 631 kB/s | 127 kB 00:00 2024-03-29T10:39:42.460 INFO:teuthology.orchestra.run.smithi088.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:39:42.461 INFO:teuthology.orchestra.run.smithi088.stdout:Total 625 kB/s | 127 kB 00:00 2024-03-29T10:39:42.461 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction check 2024-03-29T10:39:42.466 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction check succeeded. 2024-03-29T10:39:42.466 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction test 2024-03-29T10:39:42.523 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction test succeeded. 2024-03-29T10:39:42.523 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction 2024-03-29T10:39:42.672 INFO:teuthology.orchestra.run.smithi088.stdout: Preparing : 1/1 2024-03-29T10:39:42.716 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:01:38 ago on Fri 29 Mar 2024 10:38:04 AM UTC. 2024-03-29T10:39:42.783 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:39:42.813 INFO:teuthology.orchestra.run.smithi196.stdout:Package librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:39:42.865 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-03-29T10:39:42.865 INFO:teuthology.orchestra.run.smithi196.stdout:Nothing to do. 2024-03-29T10:39:42.866 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-03-29T10:39:42.913 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install librbd1 2024-03-29T10:39:43.261 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:39:43.481 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:01:39 ago on Fri 29 Mar 2024 10:38:04 AM UTC. 2024-03-29T10:39:43.578 INFO:teuthology.orchestra.run.smithi196.stdout:Package librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:39:43.578 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:39:43.578 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-03-29T10:39:43.578 INFO:teuthology.orchestra.run.smithi088.stdout:Installed: 2024-03-29T10:39:43.578 INFO:teuthology.orchestra.run.smithi088.stdout: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:39:43.578 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-03-29T10:39:43.579 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2024-03-29T10:39:43.631 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-03-29T10:39:43.632 INFO:teuthology.orchestra.run.smithi196.stdout:Nothing to do. 2024-03-29T10:39:43.632 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-03-29T10:39:43.680 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install python3-rados 2024-03-29T10:39:43.818 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install libcephfs2 2024-03-29T10:39:44.245 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:01:40 ago on Fri 29 Mar 2024 10:38:04 AM UTC. 2024-03-29T10:39:44.344 INFO:teuthology.orchestra.run.smithi196.stdout:Package python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:39:44.383 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:01:39 ago on Fri 29 Mar 2024 10:38:05 AM UTC. 2024-03-29T10:39:44.395 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-03-29T10:39:44.397 INFO:teuthology.orchestra.run.smithi196.stdout:Nothing to do. 2024-03-29T10:39:44.397 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-03-29T10:39:44.443 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install python3-rgw 2024-03-29T10:39:44.486 INFO:teuthology.orchestra.run.smithi088.stdout:Package libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:39:44.542 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2024-03-29T10:39:44.546 INFO:teuthology.orchestra.run.smithi088.stdout:Nothing to do. 2024-03-29T10:39:44.546 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2024-03-29T10:39:44.600 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install libcephfs-devel 2024-03-29T10:39:45.007 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:01:41 ago on Fri 29 Mar 2024 10:38:04 AM UTC. 2024-03-29T10:39:45.106 INFO:teuthology.orchestra.run.smithi196.stdout:Package python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:39:45.159 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-03-29T10:39:45.160 INFO:teuthology.orchestra.run.smithi196.stdout:Nothing to do. 2024-03-29T10:39:45.160 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-03-29T10:39:45.166 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:01:40 ago on Fri 29 Mar 2024 10:38:05 AM UTC. 2024-03-29T10:39:45.214 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install python3-cephfs 2024-03-29T10:39:45.317 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2024-03-29T10:39:45.318 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-03-29T10:39:45.318 INFO:teuthology.orchestra.run.smithi088.stdout: Package Arch Version Repo Size 2024-03-29T10:39:45.318 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-03-29T10:39:45.318 INFO:teuthology.orchestra.run.smithi088.stdout:Installing: 2024-03-29T10:39:45.319 INFO:teuthology.orchestra.run.smithi088.stdout: libcephfs-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 31 k 2024-03-29T10:39:45.319 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-03-29T10:39:45.319 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction Summary 2024-03-29T10:39:45.319 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-03-29T10:39:45.319 INFO:teuthology.orchestra.run.smithi088.stdout:Install 1 Package 2024-03-29T10:39:45.319 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-03-29T10:39:45.319 INFO:teuthology.orchestra.run.smithi088.stdout:Total download size: 31 k 2024-03-29T10:39:45.319 INFO:teuthology.orchestra.run.smithi088.stdout:Installed size: 140 k 2024-03-29T10:39:45.319 INFO:teuthology.orchestra.run.smithi088.stdout:Downloading Packages: 2024-03-29T10:39:45.488 INFO:teuthology.orchestra.run.smithi088.stdout:libcephfs-devel-19.0.0-1439.g5dcb190d.el9.x86_6 185 kB/s | 31 kB 00:00 2024-03-29T10:39:45.488 INFO:teuthology.orchestra.run.smithi088.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:39:45.489 INFO:teuthology.orchestra.run.smithi088.stdout:Total 183 kB/s | 31 kB 00:00 2024-03-29T10:39:45.489 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction check 2024-03-29T10:39:45.492 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction check succeeded. 2024-03-29T10:39:45.492 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction test 2024-03-29T10:39:45.518 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction test succeeded. 2024-03-29T10:39:45.519 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction 2024-03-29T10:39:45.643 INFO:teuthology.orchestra.run.smithi088.stdout: Preparing : 1/1 2024-03-29T10:39:45.760 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:39:45.786 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:01:41 ago on Fri 29 Mar 2024 10:38:04 AM UTC. 2024-03-29T10:39:45.885 INFO:teuthology.orchestra.run.smithi196.stdout:Package python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:39:45.937 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-03-29T10:39:45.938 INFO:teuthology.orchestra.run.smithi196.stdout:Nothing to do. 2024-03-29T10:39:45.938 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-03-29T10:39:45.997 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install python3-rbd 2024-03-29T10:39:46.166 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:39:46.428 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:39:46.429 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-03-29T10:39:46.429 INFO:teuthology.orchestra.run.smithi088.stdout:Installed: 2024-03-29T10:39:46.429 INFO:teuthology.orchestra.run.smithi088.stdout: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:39:46.429 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-03-29T10:39:46.429 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2024-03-29T10:39:46.565 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:01:42 ago on Fri 29 Mar 2024 10:38:04 AM UTC. 2024-03-29T10:39:46.602 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install librados2 2024-03-29T10:39:46.664 INFO:teuthology.orchestra.run.smithi196.stdout:Package python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:39:46.716 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-03-29T10:39:46.717 INFO:teuthology.orchestra.run.smithi196.stdout:Nothing to do. 2024-03-29T10:39:46.717 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-03-29T10:39:46.776 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install rbd-fuse 2024-03-29T10:39:47.160 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:01:42 ago on Fri 29 Mar 2024 10:38:05 AM UTC. 2024-03-29T10:39:47.257 INFO:teuthology.orchestra.run.smithi088.stdout:Package librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:39:47.309 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2024-03-29T10:39:47.310 INFO:teuthology.orchestra.run.smithi088.stdout:Nothing to do. 2024-03-29T10:39:47.310 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2024-03-29T10:39:47.340 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:01:43 ago on Fri 29 Mar 2024 10:38:04 AM UTC. 2024-03-29T10:39:47.360 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install librbd1 2024-03-29T10:39:47.489 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-03-29T10:39:47.490 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-29T10:39:47.490 INFO:teuthology.orchestra.run.smithi196.stdout: Package Architecture Version Repository Size 2024-03-29T10:39:47.490 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-29T10:39:47.490 INFO:teuthology.orchestra.run.smithi196.stdout:Installing: 2024-03-29T10:39:47.490 INFO:teuthology.orchestra.run.smithi196.stdout: rbd-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 85 k 2024-03-29T10:39:47.490 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-29T10:39:47.490 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction Summary 2024-03-29T10:39:47.490 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-29T10:39:47.491 INFO:teuthology.orchestra.run.smithi196.stdout:Install 1 Package 2024-03-29T10:39:47.491 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-29T10:39:47.491 INFO:teuthology.orchestra.run.smithi196.stdout:Total download size: 85 k 2024-03-29T10:39:47.491 INFO:teuthology.orchestra.run.smithi196.stdout:Installed size: 231 k 2024-03-29T10:39:47.491 INFO:teuthology.orchestra.run.smithi196.stdout:Downloading Packages: 2024-03-29T10:39:47.803 INFO:teuthology.orchestra.run.smithi196.stdout:rbd-fuse-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 273 kB/s | 85 kB 00:00 2024-03-29T10:39:47.803 INFO:teuthology.orchestra.run.smithi196.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:39:47.803 INFO:teuthology.orchestra.run.smithi196.stdout:Total 272 kB/s | 85 kB 00:00 2024-03-29T10:39:47.804 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction check 2024-03-29T10:39:47.810 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction check succeeded. 2024-03-29T10:39:47.810 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction test 2024-03-29T10:39:47.866 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction test succeeded. 2024-03-29T10:39:47.867 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction 2024-03-29T10:39:47.921 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:01:42 ago on Fri 29 Mar 2024 10:38:05 AM UTC. 2024-03-29T10:39:48.004 INFO:teuthology.orchestra.run.smithi196.stdout: Preparing : 1/1 2024-03-29T10:39:48.018 INFO:teuthology.orchestra.run.smithi088.stdout:Package librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:39:48.069 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2024-03-29T10:39:48.070 INFO:teuthology.orchestra.run.smithi088.stdout:Nothing to do. 2024-03-29T10:39:48.070 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2024-03-29T10:39:48.101 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:39:48.118 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install python3-rados 2024-03-29T10:39:48.466 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:39:48.679 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:01:43 ago on Fri 29 Mar 2024 10:38:05 AM UTC. 2024-03-29T10:39:48.777 INFO:teuthology.orchestra.run.smithi088.stdout:Package python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:39:48.787 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:39:48.787 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-29T10:39:48.787 INFO:teuthology.orchestra.run.smithi196.stdout:Installed: 2024-03-29T10:39:48.787 INFO:teuthology.orchestra.run.smithi196.stdout: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:39:48.787 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-29T10:39:48.787 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-03-29T10:39:48.829 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2024-03-29T10:39:48.830 INFO:teuthology.orchestra.run.smithi088.stdout:Nothing to do. 2024-03-29T10:39:48.830 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2024-03-29T10:39:48.874 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install python3-rgw 2024-03-29T10:39:49.046 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install rbd-mirror 2024-03-29T10:39:49.437 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:01:44 ago on Fri 29 Mar 2024 10:38:05 AM UTC. 2024-03-29T10:39:49.535 INFO:teuthology.orchestra.run.smithi088.stdout:Package python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:39:49.588 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2024-03-29T10:39:49.589 INFO:teuthology.orchestra.run.smithi088.stdout:Nothing to do. 2024-03-29T10:39:49.589 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2024-03-29T10:39:49.635 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install python3-cephfs 2024-03-29T10:39:49.643 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:01:45 ago on Fri 29 Mar 2024 10:38:04 AM UTC. 2024-03-29T10:39:49.801 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-03-29T10:39:49.802 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-29T10:39:49.802 INFO:teuthology.orchestra.run.smithi196.stdout: Package Arch Version Repo Size 2024-03-29T10:39:49.802 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-29T10:39:49.802 INFO:teuthology.orchestra.run.smithi196.stdout:Installing: 2024-03-29T10:39:49.802 INFO:teuthology.orchestra.run.smithi196.stdout: rbd-mirror x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-29T10:39:49.802 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-29T10:39:49.802 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction Summary 2024-03-29T10:39:49.802 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-29T10:39:49.802 INFO:teuthology.orchestra.run.smithi196.stdout:Install 1 Package 2024-03-29T10:39:49.803 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-29T10:39:49.803 INFO:teuthology.orchestra.run.smithi196.stdout:Total download size: 3.1 M 2024-03-29T10:39:49.803 INFO:teuthology.orchestra.run.smithi196.stdout:Installed size: 13 M 2024-03-29T10:39:49.803 INFO:teuthology.orchestra.run.smithi196.stdout:Downloading Packages: 2024-03-29T10:39:50.195 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:01:45 ago on Fri 29 Mar 2024 10:38:05 AM UTC. 2024-03-29T10:39:50.293 INFO:teuthology.orchestra.run.smithi088.stdout:Package python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:39:50.345 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2024-03-29T10:39:50.345 INFO:teuthology.orchestra.run.smithi088.stdout:Nothing to do. 2024-03-29T10:39:50.346 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2024-03-29T10:39:50.390 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install python3-rbd 2024-03-29T10:39:50.423 INFO:teuthology.orchestra.run.smithi196.stdout:rbd-mirror-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 5.0 MB/s | 3.1 MB 00:00 2024-03-29T10:39:50.424 INFO:teuthology.orchestra.run.smithi196.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:39:50.424 INFO:teuthology.orchestra.run.smithi196.stdout:Total 5.0 MB/s | 3.1 MB 00:00 2024-03-29T10:39:50.425 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction check 2024-03-29T10:39:50.434 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction check succeeded. 2024-03-29T10:39:50.434 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction test 2024-03-29T10:39:50.508 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction test succeeded. 2024-03-29T10:39:50.508 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction 2024-03-29T10:39:50.938 INFO:teuthology.orchestra.run.smithi196.stdout: Preparing : 1/1 2024-03-29T10:39:50.953 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:01:45 ago on Fri 29 Mar 2024 10:38:05 AM UTC. 2024-03-29T10:39:50.999 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:39:51.025 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:39:51.025 INFO:teuthology.orchestra.run.smithi196.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:39:51.025 INFO:teuthology.orchestra.run.smithi196.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-29T10:39:51.025 INFO:teuthology.orchestra.run.smithi196.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-29T10:39:51.025 INFO:teuthology.orchestra.run.smithi196.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-29T10:39:51.025 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-29T10:39:51.050 INFO:teuthology.orchestra.run.smithi088.stdout:Package python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:39:51.102 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2024-03-29T10:39:51.103 INFO:teuthology.orchestra.run.smithi088.stdout:Nothing to do. 2024-03-29T10:39:51.103 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2024-03-29T10:39:51.148 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install rbd-fuse 2024-03-29T10:39:51.714 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:01:46 ago on Fri 29 Mar 2024 10:38:05 AM UTC. 2024-03-29T10:39:51.862 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2024-03-29T10:39:51.862 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-03-29T10:39:51.863 INFO:teuthology.orchestra.run.smithi088.stdout: Package Architecture Version Repository Size 2024-03-29T10:39:51.863 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-03-29T10:39:51.863 INFO:teuthology.orchestra.run.smithi088.stdout:Installing: 2024-03-29T10:39:51.863 INFO:teuthology.orchestra.run.smithi088.stdout: rbd-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 85 k 2024-03-29T10:39:51.863 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-03-29T10:39:51.863 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction Summary 2024-03-29T10:39:51.863 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-03-29T10:39:51.863 INFO:teuthology.orchestra.run.smithi088.stdout:Install 1 Package 2024-03-29T10:39:51.863 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-03-29T10:39:51.864 INFO:teuthology.orchestra.run.smithi088.stdout:Total download size: 85 k 2024-03-29T10:39:51.864 INFO:teuthology.orchestra.run.smithi088.stdout:Installed size: 231 k 2024-03-29T10:39:51.864 INFO:teuthology.orchestra.run.smithi088.stdout:Downloading Packages: 2024-03-29T10:39:52.056 INFO:teuthology.orchestra.run.smithi088.stdout:rbd-fuse-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 442 kB/s | 85 kB 00:00 2024-03-29T10:39:52.057 INFO:teuthology.orchestra.run.smithi088.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:39:52.057 INFO:teuthology.orchestra.run.smithi088.stdout:Total 439 kB/s | 85 kB 00:00 2024-03-29T10:39:52.057 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction check 2024-03-29T10:39:52.064 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction check succeeded. 2024-03-29T10:39:52.064 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction test 2024-03-29T10:39:52.119 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction test succeeded. 2024-03-29T10:39:52.119 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction 2024-03-29T10:39:52.230 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:39:52.230 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-29T10:39:52.230 INFO:teuthology.orchestra.run.smithi196.stdout:Installed: 2024-03-29T10:39:52.230 INFO:teuthology.orchestra.run.smithi196.stdout: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:39:52.230 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-29T10:39:52.230 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-03-29T10:39:52.271 INFO:teuthology.orchestra.run.smithi088.stdout: Preparing : 1/1 2024-03-29T10:39:52.363 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:39:52.529 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install rbd-nbd 2024-03-29T10:39:52.744 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:39:53.022 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:39:53.023 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-03-29T10:39:53.023 INFO:teuthology.orchestra.run.smithi088.stdout:Installed: 2024-03-29T10:39:53.023 INFO:teuthology.orchestra.run.smithi088.stdout: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:39:53.023 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-03-29T10:39:53.023 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2024-03-29T10:39:53.104 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:01:49 ago on Fri 29 Mar 2024 10:38:04 AM UTC. 2024-03-29T10:39:53.254 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-03-29T10:39:53.254 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-29T10:39:53.255 INFO:teuthology.orchestra.run.smithi196.stdout: Package Architecture Version Repository Size 2024-03-29T10:39:53.255 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-29T10:39:53.255 INFO:teuthology.orchestra.run.smithi196.stdout:Installing: 2024-03-29T10:39:53.255 INFO:teuthology.orchestra.run.smithi196.stdout: rbd-nbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 171 k 2024-03-29T10:39:53.255 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-29T10:39:53.255 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction Summary 2024-03-29T10:39:53.255 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-29T10:39:53.255 INFO:teuthology.orchestra.run.smithi196.stdout:Install 1 Package 2024-03-29T10:39:53.255 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-29T10:39:53.255 INFO:teuthology.orchestra.run.smithi196.stdout:Total download size: 171 k 2024-03-29T10:39:53.256 INFO:teuthology.orchestra.run.smithi196.stdout:Installed size: 494 k 2024-03-29T10:39:53.256 INFO:teuthology.orchestra.run.smithi196.stdout:Downloading Packages: 2024-03-29T10:39:53.257 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install rbd-mirror 2024-03-29T10:39:53.470 INFO:teuthology.orchestra.run.smithi196.stdout:rbd-nbd-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 801 kB/s | 171 kB 00:00 2024-03-29T10:39:53.470 INFO:teuthology.orchestra.run.smithi196.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:39:53.470 INFO:teuthology.orchestra.run.smithi196.stdout:Total 796 kB/s | 171 kB 00:00 2024-03-29T10:39:53.470 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction check 2024-03-29T10:39:53.478 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction check succeeded. 2024-03-29T10:39:53.478 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction test 2024-03-29T10:39:53.534 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction test succeeded. 2024-03-29T10:39:53.534 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction 2024-03-29T10:39:53.678 INFO:teuthology.orchestra.run.smithi196.stdout: Preparing : 1/1 2024-03-29T10:39:53.776 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:39:53.831 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:01:48 ago on Fri 29 Mar 2024 10:38:05 AM UTC. 2024-03-29T10:39:53.993 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2024-03-29T10:39:53.994 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-03-29T10:39:53.994 INFO:teuthology.orchestra.run.smithi088.stdout: Package Arch Version Repo Size 2024-03-29T10:39:53.994 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-03-29T10:39:53.994 INFO:teuthology.orchestra.run.smithi088.stdout:Installing: 2024-03-29T10:39:53.994 INFO:teuthology.orchestra.run.smithi088.stdout: rbd-mirror x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-29T10:39:53.994 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-03-29T10:39:53.994 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction Summary 2024-03-29T10:39:53.994 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-03-29T10:39:53.994 INFO:teuthology.orchestra.run.smithi088.stdout:Install 1 Package 2024-03-29T10:39:53.994 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-03-29T10:39:53.995 INFO:teuthology.orchestra.run.smithi088.stdout:Total download size: 3.1 M 2024-03-29T10:39:53.995 INFO:teuthology.orchestra.run.smithi088.stdout:Installed size: 13 M 2024-03-29T10:39:53.995 INFO:teuthology.orchestra.run.smithi088.stdout:Downloading Packages: 2024-03-29T10:39:54.205 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:39:54.497 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:39:54.497 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-29T10:39:54.497 INFO:teuthology.orchestra.run.smithi196.stdout:Installed: 2024-03-29T10:39:54.497 INFO:teuthology.orchestra.run.smithi196.stdout: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:39:54.497 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-29T10:39:54.497 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-03-29T10:39:54.601 INFO:teuthology.orchestra.run.smithi088.stdout:rbd-mirror-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 5.1 MB/s | 3.1 MB 00:00 2024-03-29T10:39:54.602 INFO:teuthology.orchestra.run.smithi088.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:39:54.602 INFO:teuthology.orchestra.run.smithi088.stdout:Total 5.1 MB/s | 3.1 MB 00:00 2024-03-29T10:39:54.602 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction check 2024-03-29T10:39:54.612 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction check succeeded. 2024-03-29T10:39:54.612 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction test 2024-03-29T10:39:54.689 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction test succeeded. 2024-03-29T10:39:54.689 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction 2024-03-29T10:39:54.756 DEBUG:teuthology.parallel:result is None 2024-03-29T10:39:55.127 INFO:teuthology.orchestra.run.smithi088.stdout: Preparing : 1/1 2024-03-29T10:39:55.194 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:39:55.220 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:39:55.220 INFO:teuthology.orchestra.run.smithi088.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:39:55.220 INFO:teuthology.orchestra.run.smithi088.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-29T10:39:55.220 INFO:teuthology.orchestra.run.smithi088.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-29T10:39:55.220 INFO:teuthology.orchestra.run.smithi088.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-29T10:39:55.220 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-03-29T10:39:56.408 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:39:56.409 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-03-29T10:39:56.409 INFO:teuthology.orchestra.run.smithi088.stdout:Installed: 2024-03-29T10:39:56.409 INFO:teuthology.orchestra.run.smithi088.stdout: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:39:56.409 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-03-29T10:39:56.409 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2024-03-29T10:39:56.658 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install rbd-nbd 2024-03-29T10:39:57.236 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:01:52 ago on Fri 29 Mar 2024 10:38:05 AM UTC. 2024-03-29T10:39:57.387 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2024-03-29T10:39:57.388 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-03-29T10:39:57.388 INFO:teuthology.orchestra.run.smithi088.stdout: Package Architecture Version Repository Size 2024-03-29T10:39:57.388 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-03-29T10:39:57.388 INFO:teuthology.orchestra.run.smithi088.stdout:Installing: 2024-03-29T10:39:57.388 INFO:teuthology.orchestra.run.smithi088.stdout: rbd-nbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 171 k 2024-03-29T10:39:57.388 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-03-29T10:39:57.388 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction Summary 2024-03-29T10:39:57.388 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-03-29T10:39:57.388 INFO:teuthology.orchestra.run.smithi088.stdout:Install 1 Package 2024-03-29T10:39:57.388 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-03-29T10:39:57.388 INFO:teuthology.orchestra.run.smithi088.stdout:Total download size: 171 k 2024-03-29T10:39:57.389 INFO:teuthology.orchestra.run.smithi088.stdout:Installed size: 494 k 2024-03-29T10:39:57.389 INFO:teuthology.orchestra.run.smithi088.stdout:Downloading Packages: 2024-03-29T10:39:57.604 INFO:teuthology.orchestra.run.smithi088.stdout:rbd-nbd-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 795 kB/s | 171 kB 00:00 2024-03-29T10:39:57.605 INFO:teuthology.orchestra.run.smithi088.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:39:57.605 INFO:teuthology.orchestra.run.smithi088.stdout:Total 789 kB/s | 171 kB 00:00 2024-03-29T10:39:57.605 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction check 2024-03-29T10:39:57.613 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction check succeeded. 2024-03-29T10:39:57.613 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction test 2024-03-29T10:39:57.670 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction test succeeded. 2024-03-29T10:39:57.670 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction 2024-03-29T10:39:57.815 INFO:teuthology.orchestra.run.smithi088.stdout: Preparing : 1/1 2024-03-29T10:39:57.919 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:39:58.346 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:39:58.628 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:39:58.628 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-03-29T10:39:58.628 INFO:teuthology.orchestra.run.smithi088.stdout:Installed: 2024-03-29T10:39:58.628 INFO:teuthology.orchestra.run.smithi088.stdout: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:39:58.628 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-03-29T10:39:58.629 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2024-03-29T10:39:58.849 DEBUG:teuthology.parallel:result is None 2024-03-29T10:39:58.849 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-29T10:39:58.991 DEBUG:teuthology.orchestra.run.smithi088:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-29T10:39:59.011 INFO:teuthology.orchestra.run.smithi088.stdout:19.0.0-1439.g5dcb190d.el9 2024-03-29T10:39:59.011 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1439.g5dcb190d.el9 2024-03-29T10:39:59.012 INFO:teuthology.task.install:The correct ceph version 19.0.0-1439.g5dcb190d is installed. 2024-03-29T10:39:59.013 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-29T10:39:59.153 DEBUG:teuthology.orchestra.run.smithi196:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-29T10:39:59.178 INFO:teuthology.orchestra.run.smithi196.stdout:19.0.0-1439.g5dcb190d.el9 2024-03-29T10:39:59.178 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1439.g5dcb190d.el9 2024-03-29T10:39:59.178 INFO:teuthology.task.install:The correct ceph version 19.0.0-1439.g5dcb190d is installed. 2024-03-29T10:39:59.180 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-29T10:39:59.180 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-03-29T10:39:59.180 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-29T10:39:59.208 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2024-03-29T10:39:59.208 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-29T10:39:59.249 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-29T10:39:59.249 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-03-29T10:39:59.250 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd of=/usr/bin/daemon-helper 2024-03-29T10:39:59.276 DEBUG:teuthology.orchestra.run.smithi088:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-29T10:39:59.344 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2024-03-29T10:39:59.344 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd of=/usr/bin/daemon-helper 2024-03-29T10:39:59.379 DEBUG:teuthology.orchestra.run.smithi196:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-29T10:39:59.447 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-29T10:39:59.448 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-03-29T10:39:59.448 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-29T10:39:59.474 DEBUG:teuthology.orchestra.run.smithi088:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-29T10:39:59.541 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2024-03-29T10:39:59.541 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-29T10:39:59.570 DEBUG:teuthology.orchestra.run.smithi196:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-29T10:39:59.639 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-29T10:39:59.639 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-03-29T10:39:59.639 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd of=/usr/bin/stdin-killer 2024-03-29T10:39:59.666 DEBUG:teuthology.orchestra.run.smithi088:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-29T10:39:59.733 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2024-03-29T10:39:59.733 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd of=/usr/bin/stdin-killer 2024-03-29T10:39:59.761 DEBUG:teuthology.orchestra.run.smithi196:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-29T10:39:59.830 INFO:teuthology.run_tasks:Running task ceph... 2024-03-29T10:39:59.921 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-03-29T10:39:59.921 DEBUG:teuthology.orchestra.run.smithi088:> sudo chmod 777 /var/log/ceph 2024-03-29T10:39:59.924 DEBUG:teuthology.orchestra.run.smithi196:> sudo chmod 777 /var/log/ceph 2024-03-29T10:39:59.953 INFO:tasks.ceph:Disabling ceph logrotate... 2024-03-29T10:39:59.953 DEBUG:teuthology.orchestra.run.smithi088:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-29T10:39:59.992 DEBUG:teuthology.orchestra.run.smithi196:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-29T10:40:00.021 INFO:tasks.ceph:Creating extra log directories... 2024-03-29T10:40:00.021 DEBUG:teuthology.orchestra.run.smithi088:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-29T10:40:00.059 DEBUG:teuthology.orchestra.run.smithi196:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-29T10:40:00.099 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-03-29T10:40:00.099 INFO:tasks.ceph:config {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'lockdep': True, 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zstd', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': False, 'mon_bind_addrvec': True} 2024-03-29T10:40:00.099 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622648', 'branch': 'main', 'description': 'rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zstd rados tasks/rados_api_tests validater/lockdep}', 'email': None, 'first_in_suite': False, 'job_id': '7622648', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 10}}, {'volumes': {'count': 4, 'size': 10}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'main'}, 'ceph': {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'lockdep': True, 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zstd', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request'], 'mon_bind_msgr2': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'main', 'sha1': '1fb5d10608dccee73edd6ef8a2c7b804a097c324'}}, 'owner': 'scheduled_amathuri@teuthology', 'priority': 150, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'], ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b']], 'seed': 8152, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'sleep_before_teardown': 0, 'subset': '99/1000', 'suite': 'rados:verify', 'suite_branch': 'main', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': '1fb5d10608dccee73edd6ef8a2c7b804a097c324', 'targets': {'smithi088.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJC0hr90S7oLfAoelGBHu4/0U2rkf61hfisamgYRLn7TC82DQRaG4HpL3kXnRcujlwQVB8jMOFZCfEWpvvZDsdI=', 'smithi196.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOmc8TxTRpU1Lg3jXUaU3i4KUHAnuTYSGL8ZhlCPHEUiKLNgxAitJvjwBVcGhpvmK8w1uOgNB2wJREwIlBz5vvs='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'workunit': {'clients': {'client.0': ['rados/test.sh']}, 'env': {'ALLOW_TIMEOUTS': '1'}, 'timeout': '6h'}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'e691533f9cbb33d85b2187bba20d7102f098636d', 'timestamp': '2024-03-26_05:45:44', 'tube': 'smithi', 'user': 'amathuri', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-03-29T10:40:00.100 DEBUG:teuthology.orchestra.run.smithi088:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-29T10:40:00.127 DEBUG:teuthology.orchestra.run.smithi196:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-29T10:40:00.144 DEBUG:teuthology.orchestra.run.smithi088:> sudo install -d -m0777 -- /var/run/ceph 2024-03-29T10:40:00.182 DEBUG:teuthology.orchestra.run.smithi196:> sudo install -d -m0777 -- /var/run/ceph 2024-03-29T10:40:00.210 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-03-29T10:40:00.210 DEBUG:teuthology.orchestra.run.smithi088:> dd if=/scratch_devs of=/dev/stdout 2024-03-29T10:40:00.262 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-29T10:40:00.262 DEBUG:teuthology.orchestra.run.smithi088:> stat /dev/vg_nvme/lv_1 2024-03-29T10:40:00.316 INFO:teuthology.orchestra.run.smithi088.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-29T10:40:00.316 INFO:teuthology.orchestra.run.smithi088.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T10:40:00.316 INFO:teuthology.orchestra.run.smithi088.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-03-29T10:40:00.316 INFO:teuthology.orchestra.run.smithi088.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T10:40:00.317 INFO:teuthology.orchestra.run.smithi088.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T10:40:00.317 INFO:teuthology.orchestra.run.smithi088.stdout:Access: 2024-03-29 10:39:55.498697326 +0000 2024-03-29T10:40:00.317 INFO:teuthology.orchestra.run.smithi088.stdout:Modify: 2024-03-29 10:37:18.963509649 +0000 2024-03-29T10:40:00.317 INFO:teuthology.orchestra.run.smithi088.stdout:Change: 2024-03-29 10:37:18.963509649 +0000 2024-03-29T10:40:00.317 INFO:teuthology.orchestra.run.smithi088.stdout: Birth: 2024-03-29 10:37:18.963509649 +0000 2024-03-29T10:40:00.317 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-29T10:40:00.382 INFO:teuthology.orchestra.run.smithi088.stderr:1+0 records in 2024-03-29T10:40:00.382 INFO:teuthology.orchestra.run.smithi088.stderr:1+0 records out 2024-03-29T10:40:00.382 INFO:teuthology.orchestra.run.smithi088.stderr:512 bytes copied, 0.000235227 s, 2.2 MB/s 2024-03-29T10:40:00.383 DEBUG:teuthology.orchestra.run.smithi088:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-29T10:40:00.439 DEBUG:teuthology.orchestra.run.smithi088:> stat /dev/vg_nvme/lv_2 2024-03-29T10:40:00.494 INFO:teuthology.orchestra.run.smithi088.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-29T10:40:00.495 INFO:teuthology.orchestra.run.smithi088.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T10:40:00.495 INFO:teuthology.orchestra.run.smithi088.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-03-29T10:40:00.495 INFO:teuthology.orchestra.run.smithi088.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T10:40:00.495 INFO:teuthology.orchestra.run.smithi088.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T10:40:00.495 INFO:teuthology.orchestra.run.smithi088.stdout:Access: 2024-03-29 10:39:55.498697326 +0000 2024-03-29T10:40:00.495 INFO:teuthology.orchestra.run.smithi088.stdout:Modify: 2024-03-29 10:37:19.295502878 +0000 2024-03-29T10:40:00.495 INFO:teuthology.orchestra.run.smithi088.stdout:Change: 2024-03-29 10:37:19.295502878 +0000 2024-03-29T10:40:00.495 INFO:teuthology.orchestra.run.smithi088.stdout: Birth: 2024-03-29 10:37:19.295502878 +0000 2024-03-29T10:40:00.495 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-29T10:40:00.559 INFO:teuthology.orchestra.run.smithi088.stderr:1+0 records in 2024-03-29T10:40:00.560 INFO:teuthology.orchestra.run.smithi088.stderr:1+0 records out 2024-03-29T10:40:00.560 INFO:teuthology.orchestra.run.smithi088.stderr:512 bytes copied, 0.000184988 s, 2.8 MB/s 2024-03-29T10:40:00.561 DEBUG:teuthology.orchestra.run.smithi088:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-29T10:40:00.616 DEBUG:teuthology.orchestra.run.smithi088:> stat /dev/vg_nvme/lv_3 2024-03-29T10:40:00.671 INFO:teuthology.orchestra.run.smithi088.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-29T10:40:00.671 INFO:teuthology.orchestra.run.smithi088.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T10:40:00.671 INFO:teuthology.orchestra.run.smithi088.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-03-29T10:40:00.671 INFO:teuthology.orchestra.run.smithi088.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T10:40:00.672 INFO:teuthology.orchestra.run.smithi088.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T10:40:00.672 INFO:teuthology.orchestra.run.smithi088.stdout:Access: 2024-03-29 10:39:55.498697326 +0000 2024-03-29T10:40:00.672 INFO:teuthology.orchestra.run.smithi088.stdout:Modify: 2024-03-29 10:37:19.642495800 +0000 2024-03-29T10:40:00.672 INFO:teuthology.orchestra.run.smithi088.stdout:Change: 2024-03-29 10:37:19.642495800 +0000 2024-03-29T10:40:00.672 INFO:teuthology.orchestra.run.smithi088.stdout: Birth: 2024-03-29 10:37:19.642495800 +0000 2024-03-29T10:40:00.672 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-29T10:40:00.736 INFO:teuthology.orchestra.run.smithi088.stderr:1+0 records in 2024-03-29T10:40:00.737 INFO:teuthology.orchestra.run.smithi088.stderr:1+0 records out 2024-03-29T10:40:00.737 INFO:teuthology.orchestra.run.smithi088.stderr:512 bytes copied, 0.000185816 s, 2.8 MB/s 2024-03-29T10:40:00.738 DEBUG:teuthology.orchestra.run.smithi088:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-29T10:40:00.794 DEBUG:teuthology.orchestra.run.smithi088:> stat /dev/vg_nvme/lv_4 2024-03-29T10:40:00.849 INFO:teuthology.orchestra.run.smithi088.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-29T10:40:00.850 INFO:teuthology.orchestra.run.smithi088.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T10:40:00.850 INFO:teuthology.orchestra.run.smithi088.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-03-29T10:40:00.850 INFO:teuthology.orchestra.run.smithi088.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T10:40:00.850 INFO:teuthology.orchestra.run.smithi088.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T10:40:00.850 INFO:teuthology.orchestra.run.smithi088.stdout:Access: 2024-03-29 10:39:55.499697306 +0000 2024-03-29T10:40:00.850 INFO:teuthology.orchestra.run.smithi088.stdout:Modify: 2024-03-29 10:37:19.995488598 +0000 2024-03-29T10:40:00.850 INFO:teuthology.orchestra.run.smithi088.stdout:Change: 2024-03-29 10:37:19.995488598 +0000 2024-03-29T10:40:00.850 INFO:teuthology.orchestra.run.smithi088.stdout: Birth: 2024-03-29 10:37:19.995488598 +0000 2024-03-29T10:40:00.851 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-29T10:40:00.915 INFO:teuthology.orchestra.run.smithi088.stderr:1+0 records in 2024-03-29T10:40:00.915 INFO:teuthology.orchestra.run.smithi088.stderr:1+0 records out 2024-03-29T10:40:00.915 INFO:teuthology.orchestra.run.smithi088.stderr:512 bytes copied, 0.000204056 s, 2.5 MB/s 2024-03-29T10:40:00.916 DEBUG:teuthology.orchestra.run.smithi088:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-29T10:40:00.972 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T10:40:00.972 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2024-03-29T10:40:00.972 DEBUG:teuthology.orchestra.run.smithi196:> dd if=/scratch_devs of=/dev/stdout 2024-03-29T10:40:00.989 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-29T10:40:00.990 DEBUG:teuthology.orchestra.run.smithi196:> stat /dev/vg_nvme/lv_1 2024-03-29T10:40:01.046 INFO:teuthology.orchestra.run.smithi196.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-29T10:40:01.046 INFO:teuthology.orchestra.run.smithi196.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T10:40:01.046 INFO:teuthology.orchestra.run.smithi196.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-03-29T10:40:01.046 INFO:teuthology.orchestra.run.smithi196.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T10:40:01.046 INFO:teuthology.orchestra.run.smithi196.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T10:40:01.046 INFO:teuthology.orchestra.run.smithi196.stdout:Access: 2024-03-29 10:39:51.282523472 +0000 2024-03-29T10:40:01.046 INFO:teuthology.orchestra.run.smithi196.stdout:Modify: 2024-03-29 10:37:18.851678854 +0000 2024-03-29T10:40:01.046 INFO:teuthology.orchestra.run.smithi196.stdout:Change: 2024-03-29 10:37:18.851678854 +0000 2024-03-29T10:40:01.046 INFO:teuthology.orchestra.run.smithi196.stdout: Birth: 2024-03-29 10:37:18.851678854 +0000 2024-03-29T10:40:01.047 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-29T10:40:01.115 INFO:teuthology.orchestra.run.smithi196.stderr:1+0 records in 2024-03-29T10:40:01.115 INFO:teuthology.orchestra.run.smithi196.stderr:1+0 records out 2024-03-29T10:40:01.115 INFO:teuthology.orchestra.run.smithi196.stderr:512 bytes copied, 0.000306414 s, 1.7 MB/s 2024-03-29T10:40:01.117 DEBUG:teuthology.orchestra.run.smithi196:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-29T10:40:01.175 DEBUG:teuthology.orchestra.run.smithi196:> stat /dev/vg_nvme/lv_2 2024-03-29T10:40:01.231 INFO:teuthology.orchestra.run.smithi196.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-29T10:40:01.231 INFO:teuthology.orchestra.run.smithi196.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T10:40:01.231 INFO:teuthology.orchestra.run.smithi196.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-03-29T10:40:01.231 INFO:teuthology.orchestra.run.smithi196.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T10:40:01.231 INFO:teuthology.orchestra.run.smithi196.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T10:40:01.231 INFO:teuthology.orchestra.run.smithi196.stdout:Access: 2024-03-29 10:39:51.282523472 +0000 2024-03-29T10:40:01.231 INFO:teuthology.orchestra.run.smithi196.stdout:Modify: 2024-03-29 10:37:19.357667806 +0000 2024-03-29T10:40:01.231 INFO:teuthology.orchestra.run.smithi196.stdout:Change: 2024-03-29 10:37:19.357667806 +0000 2024-03-29T10:40:01.231 INFO:teuthology.orchestra.run.smithi196.stdout: Birth: 2024-03-29 10:37:19.357667806 +0000 2024-03-29T10:40:01.232 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-29T10:40:01.299 INFO:teuthology.orchestra.run.smithi196.stderr:1+0 records in 2024-03-29T10:40:01.299 INFO:teuthology.orchestra.run.smithi196.stderr:1+0 records out 2024-03-29T10:40:01.299 INFO:teuthology.orchestra.run.smithi196.stderr:512 bytes copied, 0.000212383 s, 2.4 MB/s 2024-03-29T10:40:01.300 DEBUG:teuthology.orchestra.run.smithi196:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-29T10:40:01.357 DEBUG:teuthology.orchestra.run.smithi196:> stat /dev/vg_nvme/lv_3 2024-03-29T10:40:01.413 INFO:teuthology.orchestra.run.smithi196.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-29T10:40:01.414 INFO:teuthology.orchestra.run.smithi196.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T10:40:01.414 INFO:teuthology.orchestra.run.smithi196.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-03-29T10:40:01.414 INFO:teuthology.orchestra.run.smithi196.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T10:40:01.414 INFO:teuthology.orchestra.run.smithi196.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T10:40:01.414 INFO:teuthology.orchestra.run.smithi196.stdout:Access: 2024-03-29 10:39:51.283523450 +0000 2024-03-29T10:40:01.414 INFO:teuthology.orchestra.run.smithi196.stdout:Modify: 2024-03-29 10:37:19.685660646 +0000 2024-03-29T10:40:01.414 INFO:teuthology.orchestra.run.smithi196.stdout:Change: 2024-03-29 10:37:19.685660646 +0000 2024-03-29T10:40:01.414 INFO:teuthology.orchestra.run.smithi196.stdout: Birth: 2024-03-29 10:37:19.685660646 +0000 2024-03-29T10:40:01.414 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-29T10:40:01.480 INFO:teuthology.orchestra.run.smithi196.stderr:1+0 records in 2024-03-29T10:40:01.480 INFO:teuthology.orchestra.run.smithi196.stderr:1+0 records out 2024-03-29T10:40:01.480 INFO:teuthology.orchestra.run.smithi196.stderr:512 bytes copied, 0.000185021 s, 2.8 MB/s 2024-03-29T10:40:01.481 DEBUG:teuthology.orchestra.run.smithi196:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-29T10:40:01.538 DEBUG:teuthology.orchestra.run.smithi196:> stat /dev/vg_nvme/lv_4 2024-03-29T10:40:01.594 INFO:teuthology.orchestra.run.smithi196.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-29T10:40:01.594 INFO:teuthology.orchestra.run.smithi196.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T10:40:01.594 INFO:teuthology.orchestra.run.smithi196.stdout:Device: 5h/5d Inode: 725 Links: 1 2024-03-29T10:40:01.594 INFO:teuthology.orchestra.run.smithi196.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T10:40:01.594 INFO:teuthology.orchestra.run.smithi196.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T10:40:01.594 INFO:teuthology.orchestra.run.smithi196.stdout:Access: 2024-03-29 10:39:51.283523450 +0000 2024-03-29T10:40:01.594 INFO:teuthology.orchestra.run.smithi196.stdout:Modify: 2024-03-29 10:37:20.028653159 +0000 2024-03-29T10:40:01.594 INFO:teuthology.orchestra.run.smithi196.stdout:Change: 2024-03-29 10:37:20.028653159 +0000 2024-03-29T10:40:01.594 INFO:teuthology.orchestra.run.smithi196.stdout: Birth: 2024-03-29 10:37:20.028653159 +0000 2024-03-29T10:40:01.595 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-29T10:40:01.660 INFO:teuthology.orchestra.run.smithi196.stderr:1+0 records in 2024-03-29T10:40:01.660 INFO:teuthology.orchestra.run.smithi196.stderr:1+0 records out 2024-03-29T10:40:01.660 INFO:teuthology.orchestra.run.smithi196.stderr:512 bytes copied, 0.000188137 s, 2.7 MB/s 2024-03-29T10:40:01.661 DEBUG:teuthology.orchestra.run.smithi196:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-29T10:40:01.717 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T10:40:01.717 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi088.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi196.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-29T10:40:01.717 INFO:tasks.ceph:Generating config... 2024-03-29T10:40:01.718 INFO:tasks.ceph:[client] debug monc = 20 2024-03-29T10:40:01.718 INFO:tasks.ceph:[client] debug ms = 1 2024-03-29T10:40:01.719 INFO:tasks.ceph:[client] debug objecter = 20 2024-03-29T10:40:01.719 INFO:tasks.ceph:[client] debug rados = 20 2024-03-29T10:40:01.719 INFO:tasks.ceph:[global] lockdep = True 2024-03-29T10:40:01.719 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-03-29T10:40:01.719 INFO:tasks.ceph:[global] mon election default strategy = 1 2024-03-29T10:40:01.719 INFO:tasks.ceph:[global] ms bind msgr2 = False 2024-03-29T10:40:01.719 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2024-03-29T10:40:01.719 INFO:tasks.ceph:[global] ms type = async 2024-03-29T10:40:01.719 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-03-29T10:40:01.720 INFO:tasks.ceph:[mgr] debug ms = 1 2024-03-29T10:40:01.720 INFO:tasks.ceph:[mon] debug mon = 20 2024-03-29T10:40:01.720 INFO:tasks.ceph:[mon] debug ms = 1 2024-03-29T10:40:01.720 INFO:tasks.ceph:[mon] debug paxos = 20 2024-03-29T10:40:01.720 INFO:tasks.ceph:[mon] mon min osdmap epochs = 50 2024-03-29T10:40:01.720 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2024-03-29T10:40:01.720 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2024-03-29T10:40:01.720 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2024-03-29T10:40:01.720 INFO:tasks.ceph:[mon] mon scrub interval = 300 2024-03-29T10:40:01.720 INFO:tasks.ceph:[mon] mon warn on pool no app = False 2024-03-29T10:40:01.720 INFO:tasks.ceph:[mon] paxos service trim min = 10 2024-03-29T10:40:01.720 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-03-29T10:40:01.720 INFO:tasks.ceph:[osd] bluestore compression algorithm = zstd 2024-03-29T10:40:01.720 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2024-03-29T10:40:01.721 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-03-29T10:40:01.721 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2024-03-29T10:40:01.721 INFO:tasks.ceph:[osd] debug bluefs = 20 2024-03-29T10:40:01.721 INFO:tasks.ceph:[osd] debug bluestore = 20 2024-03-29T10:40:01.721 INFO:tasks.ceph:[osd] debug monc = 20 2024-03-29T10:40:01.721 INFO:tasks.ceph:[osd] debug ms = 1 2024-03-29T10:40:01.721 INFO:tasks.ceph:[osd] debug osd = 20 2024-03-29T10:40:01.721 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-03-29T10:40:01.721 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-03-29T10:40:01.721 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-03-29T10:40:01.721 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-03-29T10:40:01.721 INFO:tasks.ceph:[osd] osd class default list = * 2024-03-29T10:40:01.722 INFO:tasks.ceph:[osd] osd class load list = * 2024-03-29T10:40:01.722 INFO:tasks.ceph:[osd] osd client watch timeout = 120 2024-03-29T10:40:01.722 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2024-03-29T10:40:01.722 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2024-03-29T10:40:01.722 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-03-29T10:40:01.722 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2024-03-29T10:40:01.722 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2024-03-29T10:40:01.722 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-03-29T10:40:01.722 INFO:tasks.ceph:[osd] osd op queue = debug_random 2024-03-29T10:40:01.722 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2024-03-29T10:40:01.722 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2024-03-29T10:40:01.723 INFO:tasks.ceph:Setting up mon.a... 2024-03-29T10:40:01.723 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-03-29T10:40:01.778 INFO:teuthology.orchestra.run.smithi088.stdout:creating /etc/ceph/ceph.keyring 2024-03-29T10:40:01.781 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-03-29T10:40:01.827 DEBUG:teuthology.orchestra.run.smithi088:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-29T10:40:01.855 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '[v1:172.21.15.88:6789]'), ('mon.c', '[v1:172.21.15.88:6790]'), ('mon.b', '[v1:172.21.15.196:6789]')] 2024-03-29T10:40:01.856 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '[v1:172.21.15.88:6789],[v1:172.21.15.88:6790],[v1:172.21.15.196:6789]', 'lockdep': True, 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zstd', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'mon.a': {}, 'mon.c': {}, 'mon.b': {}} 2024-03-29T10:40:01.859 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-03-29T10:40:01.859 DEBUG:teuthology.orchestra.run.smithi088:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-03-29T10:40:01.913 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --addv a '[v1:172.21.15.88:6789]' --addv c '[v1:172.21.15.88:6790]' --addv b '[v1:172.21.15.196:6789]' --print /home/ubuntu/cephtest/ceph.monmap 2024-03-29T10:40:02.002 INFO:teuthology.orchestra.run.smithi088.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-03-29T10:40:02.002 INFO:teuthology.orchestra.run.smithi088.stdout:monmaptool: generated fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:40:02.002 INFO:teuthology.orchestra.run.smithi088.stdout:setting min_mon_release = quincy 2024-03-29T10:40:02.002 INFO:teuthology.orchestra.run.smithi088.stdout:epoch 0 2024-03-29T10:40:02.003 INFO:teuthology.orchestra.run.smithi088.stdout:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:40:02.003 INFO:teuthology.orchestra.run.smithi088.stdout:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:40:02.003 INFO:teuthology.orchestra.run.smithi088.stdout:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:40:02.003 INFO:teuthology.orchestra.run.smithi088.stdout:min_mon_release 17 (quincy) 2024-03-29T10:40:02.003 INFO:teuthology.orchestra.run.smithi088.stdout:election_strategy: 1 2024-03-29T10:40:02.003 INFO:teuthology.orchestra.run.smithi088.stdout:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:40:02.003 INFO:teuthology.orchestra.run.smithi088.stdout:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:40:02.003 INFO:teuthology.orchestra.run.smithi088.stdout:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:40:02.003 INFO:teuthology.orchestra.run.smithi088.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-03-29T10:40:02.005 DEBUG:teuthology.orchestra.run.smithi088:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-03-29T10:40:02.020 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 869f5188-f92b-42dd-a46d-8c646e0388a9... 2024-03-29T10:40:02.022 DEBUG:teuthology.orchestra.run.smithi088:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-29T10:40:02.064 DEBUG:teuthology.orchestra.run.smithi196:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-29T10:40:02.111 INFO:teuthology.orchestra.run.smithi088.stdout:[global] 2024-03-29T10:40:02.112 INFO:teuthology.orchestra.run.smithi088.stdout: chdir = "" 2024-03-29T10:40:02.112 INFO:teuthology.orchestra.run.smithi088.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-29T10:40:02.112 INFO:teuthology.orchestra.run.smithi088.stdout: auth supported = cephx 2024-03-29T10:40:02.112 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-03-29T10:40:02.112 INFO:teuthology.orchestra.run.smithi088.stdout: filestore xattr use omap = true 2024-03-29T10:40:02.112 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-03-29T10:40:02.112 INFO:teuthology.orchestra.run.smithi088.stdout: mon clock drift allowed = 1.000 2024-03-29T10:40:02.113 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-03-29T10:40:02.113 INFO:teuthology.orchestra.run.smithi088.stdout: osd crush chooseleaf type = 0 2024-03-29T10:40:02.113 INFO:teuthology.orchestra.run.smithi088.stdout: auth debug = true 2024-03-29T10:40:02.113 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-03-29T10:40:02.113 INFO:teuthology.orchestra.run.smithi088.stdout: ms die on old message = true 2024-03-29T10:40:02.113 INFO:teuthology.orchestra.run.smithi088.stdout: ms die on bug = true 2024-03-29T10:40:02.113 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-03-29T10:40:02.113 INFO:teuthology.orchestra.run.smithi088.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-29T10:40:02.114 INFO:teuthology.orchestra.run.smithi088.stdout: mon pg warn max object skew = 0 2024-03-29T10:40:02.114 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-03-29T10:40:02.114 INFO:teuthology.orchestra.run.smithi088.stdout: # disable pg_autoscaler by default for new pools 2024-03-29T10:40:02.114 INFO:teuthology.orchestra.run.smithi088.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-29T10:40:02.114 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-03-29T10:40:02.114 INFO:teuthology.orchestra.run.smithi088.stdout: osd pool default size = 2 2024-03-29T10:40:02.114 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-03-29T10:40:02.114 INFO:teuthology.orchestra.run.smithi088.stdout: mon osd allow primary affinity = true 2024-03-29T10:40:02.115 INFO:teuthology.orchestra.run.smithi088.stdout: mon osd allow pg remap = true 2024-03-29T10:40:02.115 INFO:teuthology.orchestra.run.smithi088.stdout: mon warn on legacy crush tunables = false 2024-03-29T10:40:02.115 INFO:teuthology.orchestra.run.smithi088.stdout: mon warn on crush straw calc version zero = false 2024-03-29T10:40:02.115 INFO:teuthology.orchestra.run.smithi088.stdout: mon warn on no sortbitwise = false 2024-03-29T10:40:02.115 INFO:teuthology.orchestra.run.smithi088.stdout: mon warn on osd down out interval zero = false 2024-03-29T10:40:02.115 INFO:teuthology.orchestra.run.smithi088.stdout: mon warn on too few osds = false 2024-03-29T10:40:02.115 INFO:teuthology.orchestra.run.smithi088.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-29T10:40:02.115 INFO:teuthology.orchestra.run.smithi088.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-29T10:40:02.115 INFO:teuthology.orchestra.run.smithi088.stdout: mon_allow_pool_size_one = true 2024-03-29T10:40:02.115 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-03-29T10:40:02.116 INFO:teuthology.orchestra.run.smithi088.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-29T10:40:02.116 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-03-29T10:40:02.116 INFO:teuthology.orchestra.run.smithi088.stdout: osd default data pool replay window = 5 2024-03-29T10:40:02.116 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-03-29T10:40:02.116 INFO:teuthology.orchestra.run.smithi088.stdout: mon allow pool delete = true 2024-03-29T10:40:02.116 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-03-29T10:40:02.116 INFO:teuthology.orchestra.run.smithi088.stdout: mon cluster log file level = debug 2024-03-29T10:40:02.116 INFO:teuthology.orchestra.run.smithi088.stdout: debug asserts on shutdown = true 2024-03-29T10:40:02.116 INFO:teuthology.orchestra.run.smithi088.stdout: mon health detail to clog = false 2024-03-29T10:40:02.117 INFO:teuthology.orchestra.run.smithi088.stdout: mon host = "[v1:172.21.15.88:6789],[v1:172.21.15.88:6790],[v1:172.21.15.196:6789]" 2024-03-29T10:40:02.117 INFO:teuthology.orchestra.run.smithi088.stdout: lockdep = True 2024-03-29T10:40:02.117 INFO:teuthology.orchestra.run.smithi088.stdout: mon client directed command retry = 5 2024-03-29T10:40:02.117 INFO:teuthology.orchestra.run.smithi088.stdout: mon election default strategy = 1 2024-03-29T10:40:02.117 INFO:teuthology.orchestra.run.smithi088.stdout: ms bind msgr2 = False 2024-03-29T10:40:02.117 INFO:teuthology.orchestra.run.smithi088.stdout: ms inject socket failures = 5000 2024-03-29T10:40:02.117 INFO:teuthology.orchestra.run.smithi088.stdout: ms type = async 2024-03-29T10:40:02.117 INFO:teuthology.orchestra.run.smithi088.stdout: fsid = 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:40:02.117 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-03-29T10:40:02.117 INFO:teuthology.orchestra.run.smithi088.stdout:[osd] 2024-03-29T10:40:02.117 INFO:teuthology.orchestra.run.smithi088.stdout: osd journal size = 100 2024-03-29T10:40:02.117 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-03-29T10:40:02.118 INFO:teuthology.orchestra.run.smithi088.stdout: osd scrub load threshold = 5.0 2024-03-29T10:40:02.118 INFO:teuthology.orchestra.run.smithi088.stdout: osd scrub max interval = 600 2024-03-29T10:40:02.118 INFO:teuthology.orchestra.run.smithi088.stdout: osd mclock profile = high_recovery_ops 2024-03-29T10:40:02.118 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-03-29T10:40:02.118 INFO:teuthology.orchestra.run.smithi088.stdout: osd recover clone overlap = true 2024-03-29T10:40:02.118 INFO:teuthology.orchestra.run.smithi088.stdout: osd recovery max chunk = 1048576 2024-03-29T10:40:02.118 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-03-29T10:40:02.118 INFO:teuthology.orchestra.run.smithi088.stdout: osd debug shutdown = true 2024-03-29T10:40:02.118 INFO:teuthology.orchestra.run.smithi088.stdout: osd debug op order = true 2024-03-29T10:40:02.118 INFO:teuthology.orchestra.run.smithi088.stdout: osd debug verify stray on activate = true 2024-03-29T10:40:02.118 INFO:teuthology.orchestra.run.smithi088.stdout: osd debug trim objects = true 2024-03-29T10:40:02.118 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-03-29T10:40:02.119 INFO:teuthology.orchestra.run.smithi088.stdout: osd open classes on start = true 2024-03-29T10:40:02.119 INFO:teuthology.orchestra.run.smithi088.stdout: osd debug pg log writeout = true 2024-03-29T10:40:02.119 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-03-29T10:40:02.119 INFO:teuthology.orchestra.run.smithi088.stdout: osd deep scrub update digest min age = 30 2024-03-29T10:40:02.119 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-03-29T10:40:02.119 INFO:teuthology.orchestra.run.smithi088.stdout: osd map max advance = 10 2024-03-29T10:40:02.119 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-03-29T10:40:02.119 INFO:teuthology.orchestra.run.smithi088.stdout: journal zero on create = true 2024-03-29T10:40:02.119 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-03-29T10:40:02.119 INFO:teuthology.orchestra.run.smithi088.stdout: filestore ondisk finisher threads = 3 2024-03-29T10:40:02.120 INFO:teuthology.orchestra.run.smithi088.stdout: filestore apply finisher threads = 3 2024-03-29T10:40:02.120 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-03-29T10:40:02.120 INFO:teuthology.orchestra.run.smithi088.stdout: bdev debug aio = true 2024-03-29T10:40:02.120 INFO:teuthology.orchestra.run.smithi088.stdout: osd debug misdirected ops = true 2024-03-29T10:40:02.120 INFO:teuthology.orchestra.run.smithi088.stdout: bluestore block size = 96636764160 2024-03-29T10:40:02.120 INFO:teuthology.orchestra.run.smithi088.stdout: bluestore compression algorithm = zstd 2024-03-29T10:40:02.120 INFO:teuthology.orchestra.run.smithi088.stdout: bluestore compression mode = aggressive 2024-03-29T10:40:02.120 INFO:teuthology.orchestra.run.smithi088.stdout: bluestore fsck on mount = True 2024-03-29T10:40:02.120 INFO:teuthology.orchestra.run.smithi088.stdout: bluestore zero block detection = True 2024-03-29T10:40:02.120 INFO:teuthology.orchestra.run.smithi088.stdout: debug bluefs = 20 2024-03-29T10:40:02.120 INFO:teuthology.orchestra.run.smithi088.stdout: debug bluestore = 20 2024-03-29T10:40:02.120 INFO:teuthology.orchestra.run.smithi088.stdout: debug monc = 20 2024-03-29T10:40:02.120 INFO:teuthology.orchestra.run.smithi088.stdout: debug ms = 1 2024-03-29T10:40:02.121 INFO:teuthology.orchestra.run.smithi088.stdout: debug osd = 20 2024-03-29T10:40:02.121 INFO:teuthology.orchestra.run.smithi088.stdout: debug rocksdb = 10 2024-03-29T10:40:02.121 INFO:teuthology.orchestra.run.smithi088.stdout: mon osd backfillfull_ratio = 0.85 2024-03-29T10:40:02.121 INFO:teuthology.orchestra.run.smithi088.stdout: mon osd full ratio = 0.9 2024-03-29T10:40:02.121 INFO:teuthology.orchestra.run.smithi088.stdout: mon osd nearfull ratio = 0.8 2024-03-29T10:40:02.121 INFO:teuthology.orchestra.run.smithi088.stdout: osd class default list = * 2024-03-29T10:40:02.121 INFO:teuthology.orchestra.run.smithi088.stdout: osd class load list = * 2024-03-29T10:40:02.121 INFO:teuthology.orchestra.run.smithi088.stdout: osd client watch timeout = 120 2024-03-29T10:40:02.121 INFO:teuthology.orchestra.run.smithi088.stdout: osd debug verify cached snaps = True 2024-03-29T10:40:02.121 INFO:teuthology.orchestra.run.smithi088.stdout: osd debug verify missing on start = True 2024-03-29T10:40:02.121 INFO:teuthology.orchestra.run.smithi088.stdout: osd failsafe full ratio = 0.95 2024-03-29T10:40:02.121 INFO:teuthology.orchestra.run.smithi088.stdout: osd mclock override recovery settings = True 2024-03-29T10:40:02.121 INFO:teuthology.orchestra.run.smithi088.stdout: osd objectstore = bluestore 2024-03-29T10:40:02.121 INFO:teuthology.orchestra.run.smithi088.stdout: osd op queue = debug_random 2024-03-29T10:40:02.122 INFO:teuthology.orchestra.run.smithi088.stdout: osd op queue cut off = debug_random 2024-03-29T10:40:02.122 INFO:teuthology.orchestra.run.smithi088.stdout: osd shutdown pgref assert = True 2024-03-29T10:40:02.122 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-03-29T10:40:02.122 INFO:teuthology.orchestra.run.smithi088.stdout:[mgr] 2024-03-29T10:40:02.122 INFO:teuthology.orchestra.run.smithi088.stdout: debug ms = 1 2024-03-29T10:40:02.122 INFO:teuthology.orchestra.run.smithi088.stdout: debug mgr = 20 2024-03-29T10:40:02.122 INFO:teuthology.orchestra.run.smithi088.stdout: debug mon = 20 2024-03-29T10:40:02.122 INFO:teuthology.orchestra.run.smithi088.stdout: debug auth = 20 2024-03-29T10:40:02.122 INFO:teuthology.orchestra.run.smithi088.stdout: mon reweight min pgs per osd = 4 2024-03-29T10:40:02.123 INFO:teuthology.orchestra.run.smithi088.stdout: mon reweight min bytes per osd = 10 2024-03-29T10:40:02.123 INFO:teuthology.orchestra.run.smithi088.stdout: mgr/telemetry/nag = false 2024-03-29T10:40:02.123 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-03-29T10:40:02.123 INFO:teuthology.orchestra.run.smithi088.stdout:[mon] 2024-03-29T10:40:02.123 INFO:teuthology.orchestra.run.smithi088.stdout: debug ms = 1 2024-03-29T10:40:02.123 INFO:teuthology.orchestra.run.smithi088.stdout: debug mon = 20 2024-03-29T10:40:02.123 INFO:teuthology.orchestra.run.smithi088.stdout: debug paxos = 20 2024-03-29T10:40:02.123 INFO:teuthology.orchestra.run.smithi088.stdout: debug auth = 20 2024-03-29T10:40:02.123 INFO:teuthology.orchestra.run.smithi088.stdout: mon data avail warn = 5 2024-03-29T10:40:02.124 INFO:teuthology.orchestra.run.smithi088.stdout: mon mgr mkfs grace = 240 2024-03-29T10:40:02.124 INFO:teuthology.orchestra.run.smithi088.stdout: mon reweight min pgs per osd = 4 2024-03-29T10:40:02.124 INFO:teuthology.orchestra.run.smithi088.stdout: mon osd reporter subtree level = osd 2024-03-29T10:40:02.124 INFO:teuthology.orchestra.run.smithi088.stdout: mon osd prime pg temp = true 2024-03-29T10:40:02.124 INFO:teuthology.orchestra.run.smithi088.stdout: mon reweight min bytes per osd = 10 2024-03-29T10:40:02.124 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-03-29T10:40:02.124 INFO:teuthology.orchestra.run.smithi088.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-29T10:40:02.125 INFO:teuthology.orchestra.run.smithi088.stdout: auth mon ticket ttl = 660 # 11m 2024-03-29T10:40:02.125 INFO:teuthology.orchestra.run.smithi088.stdout: auth service ticket ttl = 240 # 4m 2024-03-29T10:40:02.125 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-03-29T10:40:02.125 INFO:teuthology.orchestra.run.smithi088.stdout: # don't complain about insecure global_id in the test suite 2024-03-29T10:40:02.125 INFO:teuthology.orchestra.run.smithi088.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-29T10:40:02.125 INFO:teuthology.orchestra.run.smithi088.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-29T10:40:02.125 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-03-29T10:40:02.126 INFO:teuthology.orchestra.run.smithi088.stdout: # 1m isn't quite enough 2024-03-29T10:40:02.126 INFO:teuthology.orchestra.run.smithi088.stdout: mon_down_mkfs_grace = 2m 2024-03-29T10:40:02.126 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-03-29T10:40:02.126 INFO:teuthology.orchestra.run.smithi088.stdout: mon_warn_on_filestore_osds = false 2024-03-29T10:40:02.126 INFO:teuthology.orchestra.run.smithi088.stdout: mon min osdmap epochs = 50 2024-03-29T10:40:02.126 INFO:teuthology.orchestra.run.smithi088.stdout: mon osdmap full prune interval = 2 2024-03-29T10:40:02.126 INFO:teuthology.orchestra.run.smithi088.stdout: mon osdmap full prune min = 15 2024-03-29T10:40:02.127 INFO:teuthology.orchestra.run.smithi088.stdout: mon osdmap full prune txsize = 2 2024-03-29T10:40:02.127 INFO:teuthology.orchestra.run.smithi088.stdout: mon scrub interval = 300 2024-03-29T10:40:02.127 INFO:teuthology.orchestra.run.smithi088.stdout: mon warn on pool no app = False 2024-03-29T10:40:02.127 INFO:teuthology.orchestra.run.smithi088.stdout: paxos service trim min = 10 2024-03-29T10:40:02.127 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-03-29T10:40:02.127 INFO:teuthology.orchestra.run.smithi088.stdout:[client] 2024-03-29T10:40:02.127 INFO:teuthology.orchestra.run.smithi088.stdout: rgw cache enabled = true 2024-03-29T10:40:02.127 INFO:teuthology.orchestra.run.smithi088.stdout: rgw enable ops log = true 2024-03-29T10:40:02.127 INFO:teuthology.orchestra.run.smithi088.stdout: rgw enable usage log = true 2024-03-29T10:40:02.128 INFO:teuthology.orchestra.run.smithi088.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-29T10:40:02.128 INFO:teuthology.orchestra.run.smithi088.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-29T10:40:02.128 INFO:teuthology.orchestra.run.smithi088.stdout: debug monc = 20 2024-03-29T10:40:02.128 INFO:teuthology.orchestra.run.smithi088.stdout: debug ms = 1 2024-03-29T10:40:02.128 INFO:teuthology.orchestra.run.smithi088.stdout: debug objecter = 20 2024-03-29T10:40:02.128 INFO:teuthology.orchestra.run.smithi088.stdout: debug rados = 20 2024-03-29T10:40:02.128 INFO:teuthology.orchestra.run.smithi088.stdout:[mon.a] 2024-03-29T10:40:02.128 INFO:teuthology.orchestra.run.smithi088.stdout:[mon.c] 2024-03-29T10:40:02.128 INFO:teuthology.orchestra.run.smithi088.stdout:[mon.b] 2024-03-29T10:40:02.129 INFO:teuthology.orchestra.run.smithi196.stdout:[global] 2024-03-29T10:40:02.129 INFO:teuthology.orchestra.run.smithi196.stdout: chdir = "" 2024-03-29T10:40:02.129 INFO:teuthology.orchestra.run.smithi196.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-29T10:40:02.129 INFO:teuthology.orchestra.run.smithi196.stdout: auth supported = cephx 2024-03-29T10:40:02.130 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-29T10:40:02.130 INFO:teuthology.orchestra.run.smithi196.stdout: filestore xattr use omap = true 2024-03-29T10:40:02.130 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-29T10:40:02.130 INFO:teuthology.orchestra.run.smithi196.stdout: mon clock drift allowed = 1.000 2024-03-29T10:40:02.130 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-29T10:40:02.130 INFO:teuthology.orchestra.run.smithi196.stdout: osd crush chooseleaf type = 0 2024-03-29T10:40:02.130 INFO:teuthology.orchestra.run.smithi196.stdout: auth debug = true 2024-03-29T10:40:02.130 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-29T10:40:02.130 INFO:teuthology.orchestra.run.smithi196.stdout: ms die on old message = true 2024-03-29T10:40:02.130 INFO:teuthology.orchestra.run.smithi196.stdout: ms die on bug = true 2024-03-29T10:40:02.130 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-29T10:40:02.131 INFO:teuthology.orchestra.run.smithi196.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-29T10:40:02.131 INFO:teuthology.orchestra.run.smithi196.stdout: mon pg warn max object skew = 0 2024-03-29T10:40:02.131 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-29T10:40:02.131 INFO:teuthology.orchestra.run.smithi196.stdout: # disable pg_autoscaler by default for new pools 2024-03-29T10:40:02.131 INFO:teuthology.orchestra.run.smithi196.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-29T10:40:02.131 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-29T10:40:02.131 INFO:teuthology.orchestra.run.smithi196.stdout: osd pool default size = 2 2024-03-29T10:40:02.131 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-29T10:40:02.131 INFO:teuthology.orchestra.run.smithi196.stdout: mon osd allow primary affinity = true 2024-03-29T10:40:02.131 INFO:teuthology.orchestra.run.smithi196.stdout: mon osd allow pg remap = true 2024-03-29T10:40:02.132 INFO:teuthology.orchestra.run.smithi196.stdout: mon warn on legacy crush tunables = false 2024-03-29T10:40:02.132 INFO:teuthology.orchestra.run.smithi196.stdout: mon warn on crush straw calc version zero = false 2024-03-29T10:40:02.132 INFO:teuthology.orchestra.run.smithi196.stdout: mon warn on no sortbitwise = false 2024-03-29T10:40:02.132 INFO:teuthology.orchestra.run.smithi196.stdout: mon warn on osd down out interval zero = false 2024-03-29T10:40:02.132 INFO:teuthology.orchestra.run.smithi196.stdout: mon warn on too few osds = false 2024-03-29T10:40:02.132 INFO:teuthology.orchestra.run.smithi196.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-29T10:40:02.132 INFO:teuthology.orchestra.run.smithi196.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-29T10:40:02.132 INFO:teuthology.orchestra.run.smithi196.stdout: mon_allow_pool_size_one = true 2024-03-29T10:40:02.132 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-29T10:40:02.132 INFO:teuthology.orchestra.run.smithi196.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-29T10:40:02.132 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-29T10:40:02.132 INFO:teuthology.orchestra.run.smithi196.stdout: osd default data pool replay window = 5 2024-03-29T10:40:02.133 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-29T10:40:02.133 INFO:teuthology.orchestra.run.smithi196.stdout: mon allow pool delete = true 2024-03-29T10:40:02.133 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-29T10:40:02.133 INFO:teuthology.orchestra.run.smithi196.stdout: mon cluster log file level = debug 2024-03-29T10:40:02.133 INFO:teuthology.orchestra.run.smithi196.stdout: debug asserts on shutdown = true 2024-03-29T10:40:02.133 INFO:teuthology.orchestra.run.smithi196.stdout: mon health detail to clog = false 2024-03-29T10:40:02.134 INFO:teuthology.orchestra.run.smithi196.stdout: mon host = "[v1:172.21.15.88:6789],[v1:172.21.15.88:6790],[v1:172.21.15.196:6789]" 2024-03-29T10:40:02.134 INFO:teuthology.orchestra.run.smithi196.stdout: lockdep = True 2024-03-29T10:40:02.134 INFO:teuthology.orchestra.run.smithi196.stdout: mon client directed command retry = 5 2024-03-29T10:40:02.134 INFO:teuthology.orchestra.run.smithi196.stdout: mon election default strategy = 1 2024-03-29T10:40:02.134 INFO:teuthology.orchestra.run.smithi196.stdout: ms bind msgr2 = False 2024-03-29T10:40:02.134 INFO:teuthology.orchestra.run.smithi196.stdout: ms inject socket failures = 5000 2024-03-29T10:40:02.134 INFO:teuthology.orchestra.run.smithi196.stdout: ms type = async 2024-03-29T10:40:02.134 INFO:teuthology.orchestra.run.smithi196.stdout: fsid = 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:40:02.134 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-29T10:40:02.135 INFO:teuthology.orchestra.run.smithi196.stdout:[osd] 2024-03-29T10:40:02.135 INFO:teuthology.orchestra.run.smithi196.stdout: osd journal size = 100 2024-03-29T10:40:02.135 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-29T10:40:02.135 INFO:teuthology.orchestra.run.smithi196.stdout: osd scrub load threshold = 5.0 2024-03-29T10:40:02.135 INFO:teuthology.orchestra.run.smithi196.stdout: osd scrub max interval = 600 2024-03-29T10:40:02.135 INFO:teuthology.orchestra.run.smithi196.stdout: osd mclock profile = high_recovery_ops 2024-03-29T10:40:02.135 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-29T10:40:02.135 INFO:teuthology.orchestra.run.smithi196.stdout: osd recover clone overlap = true 2024-03-29T10:40:02.135 INFO:teuthology.orchestra.run.smithi196.stdout: osd recovery max chunk = 1048576 2024-03-29T10:40:02.135 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-29T10:40:02.135 INFO:teuthology.orchestra.run.smithi196.stdout: osd debug shutdown = true 2024-03-29T10:40:02.135 INFO:teuthology.orchestra.run.smithi196.stdout: osd debug op order = true 2024-03-29T10:40:02.136 INFO:teuthology.orchestra.run.smithi196.stdout: osd debug verify stray on activate = true 2024-03-29T10:40:02.136 INFO:teuthology.orchestra.run.smithi196.stdout: osd debug trim objects = true 2024-03-29T10:40:02.136 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-29T10:40:02.136 INFO:teuthology.orchestra.run.smithi196.stdout: osd open classes on start = true 2024-03-29T10:40:02.136 INFO:teuthology.orchestra.run.smithi196.stdout: osd debug pg log writeout = true 2024-03-29T10:40:02.136 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-29T10:40:02.136 INFO:teuthology.orchestra.run.smithi196.stdout: osd deep scrub update digest min age = 30 2024-03-29T10:40:02.136 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-29T10:40:02.136 INFO:teuthology.orchestra.run.smithi196.stdout: osd map max advance = 10 2024-03-29T10:40:02.136 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-29T10:40:02.136 INFO:teuthology.orchestra.run.smithi196.stdout: journal zero on create = true 2024-03-29T10:40:02.136 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-29T10:40:02.136 INFO:teuthology.orchestra.run.smithi196.stdout: filestore ondisk finisher threads = 3 2024-03-29T10:40:02.136 INFO:teuthology.orchestra.run.smithi196.stdout: filestore apply finisher threads = 3 2024-03-29T10:40:02.137 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-29T10:40:02.137 INFO:teuthology.orchestra.run.smithi196.stdout: bdev debug aio = true 2024-03-29T10:40:02.137 INFO:teuthology.orchestra.run.smithi196.stdout: osd debug misdirected ops = true 2024-03-29T10:40:02.137 INFO:teuthology.orchestra.run.smithi196.stdout: bluestore block size = 96636764160 2024-03-29T10:40:02.137 INFO:teuthology.orchestra.run.smithi196.stdout: bluestore compression algorithm = zstd 2024-03-29T10:40:02.137 INFO:teuthology.orchestra.run.smithi196.stdout: bluestore compression mode = aggressive 2024-03-29T10:40:02.137 INFO:teuthology.orchestra.run.smithi196.stdout: bluestore fsck on mount = True 2024-03-29T10:40:02.137 INFO:teuthology.orchestra.run.smithi196.stdout: bluestore zero block detection = True 2024-03-29T10:40:02.137 INFO:teuthology.orchestra.run.smithi196.stdout: debug bluefs = 20 2024-03-29T10:40:02.137 INFO:teuthology.orchestra.run.smithi196.stdout: debug bluestore = 20 2024-03-29T10:40:02.137 INFO:teuthology.orchestra.run.smithi196.stdout: debug monc = 20 2024-03-29T10:40:02.137 INFO:teuthology.orchestra.run.smithi196.stdout: debug ms = 1 2024-03-29T10:40:02.137 INFO:teuthology.orchestra.run.smithi196.stdout: debug osd = 20 2024-03-29T10:40:02.138 INFO:teuthology.orchestra.run.smithi196.stdout: debug rocksdb = 10 2024-03-29T10:40:02.138 INFO:teuthology.orchestra.run.smithi196.stdout: mon osd backfillfull_ratio = 0.85 2024-03-29T10:40:02.138 INFO:teuthology.orchestra.run.smithi196.stdout: mon osd full ratio = 0.9 2024-03-29T10:40:02.138 INFO:teuthology.orchestra.run.smithi196.stdout: mon osd nearfull ratio = 0.8 2024-03-29T10:40:02.138 INFO:teuthology.orchestra.run.smithi196.stdout: osd class default list = * 2024-03-29T10:40:02.138 INFO:teuthology.orchestra.run.smithi196.stdout: osd class load list = * 2024-03-29T10:40:02.138 INFO:teuthology.orchestra.run.smithi196.stdout: osd client watch timeout = 120 2024-03-29T10:40:02.138 INFO:teuthology.orchestra.run.smithi196.stdout: osd debug verify cached snaps = True 2024-03-29T10:40:02.138 INFO:teuthology.orchestra.run.smithi196.stdout: osd debug verify missing on start = True 2024-03-29T10:40:02.138 INFO:teuthology.orchestra.run.smithi196.stdout: osd failsafe full ratio = 0.95 2024-03-29T10:40:02.139 INFO:teuthology.orchestra.run.smithi196.stdout: osd mclock override recovery settings = True 2024-03-29T10:40:02.139 INFO:teuthology.orchestra.run.smithi196.stdout: osd objectstore = bluestore 2024-03-29T10:40:02.139 INFO:teuthology.orchestra.run.smithi196.stdout: osd op queue = debug_random 2024-03-29T10:40:02.139 INFO:teuthology.orchestra.run.smithi196.stdout: osd op queue cut off = debug_random 2024-03-29T10:40:02.139 INFO:teuthology.orchestra.run.smithi196.stdout: osd shutdown pgref assert = True 2024-03-29T10:40:02.139 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-29T10:40:02.139 INFO:teuthology.orchestra.run.smithi196.stdout:[mgr] 2024-03-29T10:40:02.139 INFO:teuthology.orchestra.run.smithi196.stdout: debug ms = 1 2024-03-29T10:40:02.139 INFO:teuthology.orchestra.run.smithi196.stdout: debug mgr = 20 2024-03-29T10:40:02.139 INFO:teuthology.orchestra.run.smithi196.stdout: debug mon = 20 2024-03-29T10:40:02.139 INFO:teuthology.orchestra.run.smithi196.stdout: debug auth = 20 2024-03-29T10:40:02.139 INFO:teuthology.orchestra.run.smithi196.stdout: mon reweight min pgs per osd = 4 2024-03-29T10:40:02.140 INFO:teuthology.orchestra.run.smithi196.stdout: mon reweight min bytes per osd = 10 2024-03-29T10:40:02.140 INFO:teuthology.orchestra.run.smithi196.stdout: mgr/telemetry/nag = false 2024-03-29T10:40:02.140 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-29T10:40:02.140 INFO:teuthology.orchestra.run.smithi196.stdout:[mon] 2024-03-29T10:40:02.140 INFO:teuthology.orchestra.run.smithi196.stdout: debug ms = 1 2024-03-29T10:40:02.140 INFO:teuthology.orchestra.run.smithi196.stdout: debug mon = 20 2024-03-29T10:40:02.140 INFO:teuthology.orchestra.run.smithi196.stdout: debug paxos = 20 2024-03-29T10:40:02.140 INFO:teuthology.orchestra.run.smithi196.stdout: debug auth = 20 2024-03-29T10:40:02.140 INFO:teuthology.orchestra.run.smithi196.stdout: mon data avail warn = 5 2024-03-29T10:40:02.140 INFO:teuthology.orchestra.run.smithi196.stdout: mon mgr mkfs grace = 240 2024-03-29T10:40:02.140 INFO:teuthology.orchestra.run.smithi196.stdout: mon reweight min pgs per osd = 4 2024-03-29T10:40:02.141 INFO:teuthology.orchestra.run.smithi196.stdout: mon osd reporter subtree level = osd 2024-03-29T10:40:02.141 INFO:teuthology.orchestra.run.smithi196.stdout: mon osd prime pg temp = true 2024-03-29T10:40:02.141 INFO:teuthology.orchestra.run.smithi196.stdout: mon reweight min bytes per osd = 10 2024-03-29T10:40:02.141 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-29T10:40:02.141 INFO:teuthology.orchestra.run.smithi196.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-29T10:40:02.141 INFO:teuthology.orchestra.run.smithi196.stdout: auth mon ticket ttl = 660 # 11m 2024-03-29T10:40:02.141 INFO:teuthology.orchestra.run.smithi196.stdout: auth service ticket ttl = 240 # 4m 2024-03-29T10:40:02.141 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-29T10:40:02.141 INFO:teuthology.orchestra.run.smithi196.stdout: # don't complain about insecure global_id in the test suite 2024-03-29T10:40:02.141 INFO:teuthology.orchestra.run.smithi196.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-29T10:40:02.141 INFO:teuthology.orchestra.run.smithi196.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-29T10:40:02.141 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-29T10:40:02.142 INFO:teuthology.orchestra.run.smithi196.stdout: # 1m isn't quite enough 2024-03-29T10:40:02.142 INFO:teuthology.orchestra.run.smithi196.stdout: mon_down_mkfs_grace = 2m 2024-03-29T10:40:02.142 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-29T10:40:02.142 INFO:teuthology.orchestra.run.smithi196.stdout: mon_warn_on_filestore_osds = false 2024-03-29T10:40:02.142 INFO:teuthology.orchestra.run.smithi196.stdout: mon min osdmap epochs = 50 2024-03-29T10:40:02.142 INFO:teuthology.orchestra.run.smithi196.stdout: mon osdmap full prune interval = 2 2024-03-29T10:40:02.142 INFO:teuthology.orchestra.run.smithi196.stdout: mon osdmap full prune min = 15 2024-03-29T10:40:02.142 INFO:teuthology.orchestra.run.smithi196.stdout: mon osdmap full prune txsize = 2 2024-03-29T10:40:02.142 INFO:teuthology.orchestra.run.smithi196.stdout: mon scrub interval = 300 2024-03-29T10:40:02.142 INFO:teuthology.orchestra.run.smithi196.stdout: mon warn on pool no app = False 2024-03-29T10:40:02.142 INFO:teuthology.orchestra.run.smithi196.stdout: paxos service trim min = 10 2024-03-29T10:40:02.143 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-29T10:40:02.143 INFO:teuthology.orchestra.run.smithi196.stdout:[client] 2024-03-29T10:40:02.143 INFO:teuthology.orchestra.run.smithi196.stdout: rgw cache enabled = true 2024-03-29T10:40:02.143 INFO:teuthology.orchestra.run.smithi196.stdout: rgw enable ops log = true 2024-03-29T10:40:02.143 INFO:teuthology.orchestra.run.smithi196.stdout: rgw enable usage log = true 2024-03-29T10:40:02.143 INFO:teuthology.orchestra.run.smithi196.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-29T10:40:02.143 INFO:teuthology.orchestra.run.smithi196.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-29T10:40:02.143 INFO:teuthology.orchestra.run.smithi196.stdout: debug monc = 20 2024-03-29T10:40:02.143 INFO:teuthology.orchestra.run.smithi196.stdout: debug ms = 1 2024-03-29T10:40:02.143 INFO:teuthology.orchestra.run.smithi196.stdout: debug objecter = 20 2024-03-29T10:40:02.143 INFO:teuthology.orchestra.run.smithi196.stdout: debug rados = 20 2024-03-29T10:40:02.143 INFO:teuthology.orchestra.run.smithi196.stdout:[mon.a] 2024-03-29T10:40:02.143 INFO:teuthology.orchestra.run.smithi196.stdout:[mon.c] 2024-03-29T10:40:02.143 INFO:teuthology.orchestra.run.smithi196.stdout:[mon.b] 2024-03-29T10:40:02.145 INFO:tasks.ceph:Creating admin key on mon.a... 2024-03-29T10:40:02.146 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-03-29T10:40:02.201 INFO:tasks.ceph:Copying monmap to all nodes... 2024-03-29T10:40:02.202 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-03-29T10:40:02.202 DEBUG:teuthology.orchestra.run.smithi088:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-03-29T10:40:02.218 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-03-29T10:40:02.218 DEBUG:teuthology.orchestra.run.smithi088:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-03-29T10:40:02.272 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi088.front.sepia.ceph.com 2024-03-29T10:40:02.272 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-03-29T10:40:02.272 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-29T10:40:02.272 DEBUG:teuthology.orchestra.run.smithi088:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-29T10:40:02.353 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-03-29T10:40:02.353 DEBUG:teuthology.orchestra.run.smithi088:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-29T10:40:02.410 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi196.front.sepia.ceph.com 2024-03-29T10:40:02.410 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2024-03-29T10:40:02.410 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-29T10:40:02.410 DEBUG:teuthology.orchestra.run.smithi196:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-29T10:40:02.457 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2024-03-29T10:40:02.457 DEBUG:teuthology.orchestra.run.smithi196:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-29T10:40:02.515 INFO:tasks.ceph:Setting up mon nodes... 2024-03-29T10:40:02.515 INFO:tasks.ceph:Setting up mgr nodes... 2024-03-29T10:40:02.515 DEBUG:teuthology.orchestra.run.smithi088:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-03-29T10:40:02.571 INFO:teuthology.orchestra.run.smithi088.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-03-29T10:40:02.573 DEBUG:teuthology.orchestra.run.smithi196:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-03-29T10:40:02.638 INFO:teuthology.orchestra.run.smithi196.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-03-29T10:40:02.643 INFO:tasks.ceph:Setting up mds nodes... 2024-03-29T10:40:02.643 INFO:tasks.ceph_client:Setting up client nodes... 2024-03-29T10:40:02.643 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-29T10:40:02.685 INFO:teuthology.orchestra.run.smithi088.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-03-29T10:40:02.700 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-03-29T10:40:02.745 INFO:teuthology.orchestra.run.smithi196.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-03-29T10:40:02.760 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-03-29T10:40:02.761 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi088.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi196.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-29T10:40:02.761 DEBUG:teuthology.orchestra.run.smithi088:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-03-29T10:40:02.788 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T10:40:02.789 INFO:tasks.ceph:role: osd.0 2024-03-29T10:40:02.789 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi088.front.sepia.ceph.com 2024-03-29T10:40:02.789 DEBUG:teuthology.orchestra.run.smithi088:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-29T10:40:02.863 INFO:teuthology.orchestra.run.smithi088.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T10:40:02.864 INFO:teuthology.orchestra.run.smithi088.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T10:40:02.864 INFO:teuthology.orchestra.run.smithi088.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T10:40:02.864 INFO:teuthology.orchestra.run.smithi088.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T10:40:02.864 INFO:teuthology.orchestra.run.smithi088.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T10:40:02.864 INFO:teuthology.orchestra.run.smithi088.stdout: = sunit=0 swidth=0 blks 2024-03-29T10:40:02.864 INFO:teuthology.orchestra.run.smithi088.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T10:40:02.864 INFO:teuthology.orchestra.run.smithi088.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T10:40:02.864 INFO:teuthology.orchestra.run.smithi088.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T10:40:02.864 INFO:teuthology.orchestra.run.smithi088.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T10:40:03.384 INFO:teuthology.orchestra.run.smithi088.stdout:Discarding blocks...Done. 2024-03-29T10:40:03.385 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi088.front.sepia.ceph.com -o noatime 2024-03-29T10:40:03.386 DEBUG:teuthology.orchestra.run.smithi088:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-03-29T10:40:03.502 DEBUG:teuthology.orchestra.run.smithi088:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-03-29T10:40:03.535 DEBUG:teuthology.orchestra.run.smithi088:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-03-29T10:40:03.602 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T10:40:03.602 INFO:tasks.ceph:role: osd.1 2024-03-29T10:40:03.602 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi088.front.sepia.ceph.com 2024-03-29T10:40:03.603 DEBUG:teuthology.orchestra.run.smithi088:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-29T10:40:03.676 INFO:teuthology.orchestra.run.smithi088.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T10:40:03.677 INFO:teuthology.orchestra.run.smithi088.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T10:40:03.677 INFO:teuthology.orchestra.run.smithi088.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T10:40:03.677 INFO:teuthology.orchestra.run.smithi088.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T10:40:03.677 INFO:teuthology.orchestra.run.smithi088.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T10:40:03.677 INFO:teuthology.orchestra.run.smithi088.stdout: = sunit=0 swidth=0 blks 2024-03-29T10:40:03.677 INFO:teuthology.orchestra.run.smithi088.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T10:40:03.677 INFO:teuthology.orchestra.run.smithi088.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T10:40:03.678 INFO:teuthology.orchestra.run.smithi088.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T10:40:03.678 INFO:teuthology.orchestra.run.smithi088.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T10:40:04.139 INFO:teuthology.orchestra.run.smithi088.stdout:Discarding blocks...Done. 2024-03-29T10:40:04.141 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi088.front.sepia.ceph.com -o noatime 2024-03-29T10:40:04.142 DEBUG:teuthology.orchestra.run.smithi088:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-03-29T10:40:04.235 DEBUG:teuthology.orchestra.run.smithi088:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-03-29T10:40:04.266 DEBUG:teuthology.orchestra.run.smithi088:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-03-29T10:40:04.336 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T10:40:04.336 INFO:tasks.ceph:role: osd.2 2024-03-29T10:40:04.336 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi088.front.sepia.ceph.com 2024-03-29T10:40:04.336 DEBUG:teuthology.orchestra.run.smithi088:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-29T10:40:04.411 INFO:teuthology.orchestra.run.smithi088.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T10:40:04.411 INFO:teuthology.orchestra.run.smithi088.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T10:40:04.411 INFO:teuthology.orchestra.run.smithi088.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T10:40:04.411 INFO:teuthology.orchestra.run.smithi088.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T10:40:04.411 INFO:teuthology.orchestra.run.smithi088.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T10:40:04.411 INFO:teuthology.orchestra.run.smithi088.stdout: = sunit=0 swidth=0 blks 2024-03-29T10:40:04.411 INFO:teuthology.orchestra.run.smithi088.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T10:40:04.411 INFO:teuthology.orchestra.run.smithi088.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T10:40:04.411 INFO:teuthology.orchestra.run.smithi088.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T10:40:04.412 INFO:teuthology.orchestra.run.smithi088.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T10:40:04.889 INFO:teuthology.orchestra.run.smithi088.stdout:Discarding blocks...Done. 2024-03-29T10:40:04.892 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi088.front.sepia.ceph.com -o noatime 2024-03-29T10:40:04.892 DEBUG:teuthology.orchestra.run.smithi088:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-03-29T10:40:05.022 DEBUG:teuthology.orchestra.run.smithi088:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-03-29T10:40:05.054 DEBUG:teuthology.orchestra.run.smithi088:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-03-29T10:40:05.122 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T10:40:05.122 INFO:tasks.ceph:role: osd.3 2024-03-29T10:40:05.122 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi088.front.sepia.ceph.com 2024-03-29T10:40:05.122 DEBUG:teuthology.orchestra.run.smithi088:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-29T10:40:05.195 INFO:teuthology.orchestra.run.smithi088.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T10:40:05.196 INFO:teuthology.orchestra.run.smithi088.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T10:40:05.196 INFO:teuthology.orchestra.run.smithi088.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T10:40:05.196 INFO:teuthology.orchestra.run.smithi088.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T10:40:05.196 INFO:teuthology.orchestra.run.smithi088.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T10:40:05.196 INFO:teuthology.orchestra.run.smithi088.stdout: = sunit=0 swidth=0 blks 2024-03-29T10:40:05.196 INFO:teuthology.orchestra.run.smithi088.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T10:40:05.196 INFO:teuthology.orchestra.run.smithi088.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T10:40:05.196 INFO:teuthology.orchestra.run.smithi088.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T10:40:05.196 INFO:teuthology.orchestra.run.smithi088.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T10:40:05.713 INFO:teuthology.orchestra.run.smithi088.stdout:Discarding blocks...Done. 2024-03-29T10:40:05.715 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi088.front.sepia.ceph.com -o noatime 2024-03-29T10:40:05.715 DEBUG:teuthology.orchestra.run.smithi088:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-03-29T10:40:05.856 DEBUG:teuthology.orchestra.run.smithi088:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-03-29T10:40:05.890 DEBUG:teuthology.orchestra.run.smithi088:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T10:40:05.996 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:06.012+0000 7f774439a640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-03-29T10:40:05.996 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:06.012+0000 7f774439a640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-03-29T10:40:05.996 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:06.012+0000 7f774439a640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-29T10:40:05.996 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:06.012+0000 7f774439a640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-29T10:40:05.997 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:06.012+0000 7f774439a640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-29T10:40:05.998 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:06.014+0000 7f774439a640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-03-29T10:40:08.853 DEBUG:teuthology.orchestra.run.smithi088:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-29T10:40:08.884 DEBUG:teuthology.orchestra.run.smithi088:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T10:40:08.975 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:08.990+0000 7fd3244bf640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-03-29T10:40:08.975 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:08.991+0000 7fd3244bf640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-03-29T10:40:08.975 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:08.991+0000 7fd3244bf640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-29T10:40:08.975 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:08.991+0000 7fd3244bf640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-29T10:40:08.976 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:08.991+0000 7fd3244bf640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-29T10:40:08.976 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:08.991+0000 7fd3244bf640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-03-29T10:40:11.823 DEBUG:teuthology.orchestra.run.smithi088:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-03-29T10:40:11.854 DEBUG:teuthology.orchestra.run.smithi088:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T10:40:11.948 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:11.964+0000 7f6b9660f640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-03-29T10:40:11.948 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:11.964+0000 7f6b9660f640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-03-29T10:40:11.948 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:11.964+0000 7f6b9660f640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-29T10:40:11.948 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:11.964+0000 7f6b9660f640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-29T10:40:11.948 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:11.964+0000 7f6b9660f640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-29T10:40:11.949 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:11.965+0000 7f6b9660f640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-03-29T10:40:14.797 DEBUG:teuthology.orchestra.run.smithi088:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-29T10:40:14.828 DEBUG:teuthology.orchestra.run.smithi088:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T10:40:14.918 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:14.934+0000 7fa6f0773640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-03-29T10:40:14.918 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:14.934+0000 7fa6f0773640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-03-29T10:40:14.918 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:14.934+0000 7fa6f0773640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-29T10:40:14.919 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:14.934+0000 7fa6f0773640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-29T10:40:14.919 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:14.934+0000 7fa6f0773640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-29T10:40:14.919 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:14.935+0000 7fa6f0773640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-03-29T10:40:17.773 DEBUG:teuthology.orchestra.run.smithi088:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-29T10:40:17.804 DEBUG:teuthology.orchestra.run.smithi196:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-03-29T10:40:17.836 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T10:40:17.837 INFO:tasks.ceph:role: osd.4 2024-03-29T10:40:17.837 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi196.front.sepia.ceph.com 2024-03-29T10:40:17.837 DEBUG:teuthology.orchestra.run.smithi196:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-29T10:40:17.914 INFO:teuthology.orchestra.run.smithi196.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T10:40:17.914 INFO:teuthology.orchestra.run.smithi196.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T10:40:17.914 INFO:teuthology.orchestra.run.smithi196.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T10:40:17.914 INFO:teuthology.orchestra.run.smithi196.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T10:40:17.915 INFO:teuthology.orchestra.run.smithi196.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T10:40:17.915 INFO:teuthology.orchestra.run.smithi196.stdout: = sunit=0 swidth=0 blks 2024-03-29T10:40:17.915 INFO:teuthology.orchestra.run.smithi196.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T10:40:17.915 INFO:teuthology.orchestra.run.smithi196.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T10:40:17.915 INFO:teuthology.orchestra.run.smithi196.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T10:40:17.915 INFO:teuthology.orchestra.run.smithi196.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T10:40:18.425 INFO:teuthology.orchestra.run.smithi196.stdout:Discarding blocks...Done. 2024-03-29T10:40:18.426 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi196.front.sepia.ceph.com -o noatime 2024-03-29T10:40:18.427 DEBUG:teuthology.orchestra.run.smithi196:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-03-29T10:40:18.572 DEBUG:teuthology.orchestra.run.smithi196:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-03-29T10:40:18.609 DEBUG:teuthology.orchestra.run.smithi196:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-03-29T10:40:18.676 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T10:40:18.676 INFO:tasks.ceph:role: osd.5 2024-03-29T10:40:18.677 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi196.front.sepia.ceph.com 2024-03-29T10:40:18.677 DEBUG:teuthology.orchestra.run.smithi196:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-29T10:40:18.750 INFO:teuthology.orchestra.run.smithi196.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T10:40:18.750 INFO:teuthology.orchestra.run.smithi196.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T10:40:18.751 INFO:teuthology.orchestra.run.smithi196.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T10:40:18.751 INFO:teuthology.orchestra.run.smithi196.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T10:40:18.751 INFO:teuthology.orchestra.run.smithi196.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T10:40:18.751 INFO:teuthology.orchestra.run.smithi196.stdout: = sunit=0 swidth=0 blks 2024-03-29T10:40:18.751 INFO:teuthology.orchestra.run.smithi196.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T10:40:18.751 INFO:teuthology.orchestra.run.smithi196.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T10:40:18.751 INFO:teuthology.orchestra.run.smithi196.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T10:40:18.751 INFO:teuthology.orchestra.run.smithi196.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T10:40:19.194 INFO:teuthology.orchestra.run.smithi196.stdout:Discarding blocks...Done. 2024-03-29T10:40:19.196 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi196.front.sepia.ceph.com -o noatime 2024-03-29T10:40:19.196 DEBUG:teuthology.orchestra.run.smithi196:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-03-29T10:40:19.281 DEBUG:teuthology.orchestra.run.smithi196:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-03-29T10:40:19.351 DEBUG:teuthology.orchestra.run.smithi196:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-03-29T10:40:19.419 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T10:40:19.419 INFO:tasks.ceph:role: osd.6 2024-03-29T10:40:19.419 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi196.front.sepia.ceph.com 2024-03-29T10:40:19.419 DEBUG:teuthology.orchestra.run.smithi196:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-29T10:40:19.493 INFO:teuthology.orchestra.run.smithi196.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T10:40:19.493 INFO:teuthology.orchestra.run.smithi196.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T10:40:19.493 INFO:teuthology.orchestra.run.smithi196.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T10:40:19.493 INFO:teuthology.orchestra.run.smithi196.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T10:40:19.493 INFO:teuthology.orchestra.run.smithi196.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T10:40:19.493 INFO:teuthology.orchestra.run.smithi196.stdout: = sunit=0 swidth=0 blks 2024-03-29T10:40:19.493 INFO:teuthology.orchestra.run.smithi196.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T10:40:19.493 INFO:teuthology.orchestra.run.smithi196.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T10:40:19.493 INFO:teuthology.orchestra.run.smithi196.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T10:40:19.493 INFO:teuthology.orchestra.run.smithi196.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T10:40:20.014 INFO:teuthology.orchestra.run.smithi196.stdout:Discarding blocks...Done. 2024-03-29T10:40:20.016 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi196.front.sepia.ceph.com -o noatime 2024-03-29T10:40:20.017 DEBUG:teuthology.orchestra.run.smithi196:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-03-29T10:40:20.165 DEBUG:teuthology.orchestra.run.smithi196:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-03-29T10:40:20.198 DEBUG:teuthology.orchestra.run.smithi196:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-03-29T10:40:20.265 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T10:40:20.266 INFO:tasks.ceph:role: osd.7 2024-03-29T10:40:20.266 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi196.front.sepia.ceph.com 2024-03-29T10:40:20.266 DEBUG:teuthology.orchestra.run.smithi196:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-29T10:40:20.340 INFO:teuthology.orchestra.run.smithi196.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T10:40:20.340 INFO:teuthology.orchestra.run.smithi196.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T10:40:20.340 INFO:teuthology.orchestra.run.smithi196.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T10:40:20.340 INFO:teuthology.orchestra.run.smithi196.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T10:40:20.340 INFO:teuthology.orchestra.run.smithi196.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T10:40:20.340 INFO:teuthology.orchestra.run.smithi196.stdout: = sunit=0 swidth=0 blks 2024-03-29T10:40:20.340 INFO:teuthology.orchestra.run.smithi196.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T10:40:20.340 INFO:teuthology.orchestra.run.smithi196.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T10:40:20.340 INFO:teuthology.orchestra.run.smithi196.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T10:40:20.340 INFO:teuthology.orchestra.run.smithi196.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T10:40:20.860 INFO:teuthology.orchestra.run.smithi196.stdout:Discarding blocks...Done. 2024-03-29T10:40:20.863 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi196.front.sepia.ceph.com -o noatime 2024-03-29T10:40:20.863 DEBUG:teuthology.orchestra.run.smithi196:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-03-29T10:40:20.960 DEBUG:teuthology.orchestra.run.smithi196:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-03-29T10:40:20.995 DEBUG:teuthology.orchestra.run.smithi196:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T10:40:21.096 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:21.111+0000 7f87be21e640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-03-29T10:40:21.096 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:21.111+0000 7f87be21e640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-03-29T10:40:21.097 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:21.111+0000 7f87be21e640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-29T10:40:21.097 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:21.111+0000 7f87be21e640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-29T10:40:21.097 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:21.111+0000 7f87be21e640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-29T10:40:21.097 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:21.112+0000 7f87be21e640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-03-29T10:40:23.943 DEBUG:teuthology.orchestra.run.smithi196:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-03-29T10:40:23.975 DEBUG:teuthology.orchestra.run.smithi196:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T10:40:24.066 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:24.082+0000 7f872c011640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-03-29T10:40:24.066 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:24.082+0000 7f872c011640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-03-29T10:40:24.066 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:24.082+0000 7f872c011640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-29T10:40:24.067 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:24.082+0000 7f872c011640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-29T10:40:24.067 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:24.082+0000 7f872c011640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-29T10:40:24.067 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:24.083+0000 7f872c011640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-03-29T10:40:26.903 DEBUG:teuthology.orchestra.run.smithi196:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-03-29T10:40:26.933 DEBUG:teuthology.orchestra.run.smithi196:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T10:40:27.023 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:27.039+0000 7f38a240d640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-03-29T10:40:27.024 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:27.039+0000 7f38a240d640 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-03-29T10:40:27.024 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:27.039+0000 7f38a240d640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-29T10:40:27.024 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:27.039+0000 7f38a240d640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-29T10:40:27.024 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:27.039+0000 7f38a240d640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-29T10:40:27.024 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:27.039+0000 7f38a240d640 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-03-29T10:40:29.877 DEBUG:teuthology.orchestra.run.smithi196:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-03-29T10:40:29.905 DEBUG:teuthology.orchestra.run.smithi196:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T10:40:29.992 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:30.008+0000 7f2596e8b640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-03-29T10:40:29.992 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:30.008+0000 7f2596e8b640 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-03-29T10:40:29.992 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:30.008+0000 7f2596e8b640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-29T10:40:29.992 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:30.008+0000 7f2596e8b640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-29T10:40:29.992 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:30.008+0000 7f2596e8b640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-29T10:40:29.992 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:30.008+0000 7f2596e8b640 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-03-29T10:40:32.842 DEBUG:teuthology.orchestra.run.smithi196:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-03-29T10:40:32.872 INFO:tasks.ceph:Reading keys from all nodes... 2024-03-29T10:40:32.872 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-03-29T10:40:32.872 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-03-29T10:40:32.907 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-03-29T10:40:32.907 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-03-29T10:40:32.980 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-03-29T10:40:32.980 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-03-29T10:40:33.056 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-03-29T10:40:33.056 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-03-29T10:40:33.130 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-03-29T10:40:33.130 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-03-29T10:40:33.208 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2024-03-29T10:40:33.208 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-03-29T10:40:33.238 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2024-03-29T10:40:33.238 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-03-29T10:40:33.313 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2024-03-29T10:40:33.314 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-03-29T10:40:33.389 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2024-03-29T10:40:33.389 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-03-29T10:40:33.470 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2024-03-29T10:40:33.470 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-03-29T10:40:33.542 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-03-29T10:40:33.543 DEBUG:teuthology.orchestra.run.smithi088:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-03-29T10:40:33.560 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2024-03-29T10:40:33.560 DEBUG:teuthology.orchestra.run.smithi196:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-03-29T10:40:33.603 INFO:tasks.ceph:Adding keys to all mons... 2024-03-29T10:40:33.603 DEBUG:teuthology.orchestra.run.smithi088:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-29T10:40:33.612 DEBUG:teuthology.orchestra.run.smithi196:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-29T10:40:33.658 INFO:teuthology.orchestra.run.smithi088.stdout:[mgr.y] 2024-03-29T10:40:33.658 INFO:teuthology.orchestra.run.smithi088.stdout: key = AQCCmgZmTrwWIxAAJClcgAW5xlcXBD3cjHZw/Q== 2024-03-29T10:40:33.659 INFO:teuthology.orchestra.run.smithi088.stdout:[osd.0] 2024-03-29T10:40:33.659 INFO:teuthology.orchestra.run.smithi088.stdout: key = AQCGmgZmfaHNABAAET1tUwl7FhyzxFwteB2oYw== 2024-03-29T10:40:33.659 INFO:teuthology.orchestra.run.smithi088.stdout:[osd.1] 2024-03-29T10:40:33.659 INFO:teuthology.orchestra.run.smithi088.stdout: key = AQCImgZmR1chOxAAcJRY+ZwP3JAzG+SOC+hDKQ== 2024-03-29T10:40:33.659 INFO:teuthology.orchestra.run.smithi088.stdout:[osd.2] 2024-03-29T10:40:33.659 INFO:teuthology.orchestra.run.smithi088.stdout: key = AQCLmgZmrV6FORAArl5tDJX0kIDz3KG43VfQpw== 2024-03-29T10:40:33.659 INFO:teuthology.orchestra.run.smithi088.stdout:[osd.3] 2024-03-29T10:40:33.660 INFO:teuthology.orchestra.run.smithi088.stdout: key = AQCOmgZm9mXANxAAldvBkyxpNEJCOYlOGKLJEg== 2024-03-29T10:40:33.660 INFO:teuthology.orchestra.run.smithi088.stdout:[mgr.x] 2024-03-29T10:40:33.660 INFO:teuthology.orchestra.run.smithi088.stdout: key = AQCCmgZm3eIRJxAAV+wluWTgoxHkFAneC1IJ8w== 2024-03-29T10:40:33.660 INFO:teuthology.orchestra.run.smithi088.stdout:[osd.4] 2024-03-29T10:40:33.660 INFO:teuthology.orchestra.run.smithi088.stdout: key = AQCVmgZmq4rCBhAAvAx2dbSZZyxglekYKq/85g== 2024-03-29T10:40:33.660 INFO:teuthology.orchestra.run.smithi088.stdout:[osd.5] 2024-03-29T10:40:33.660 INFO:teuthology.orchestra.run.smithi088.stdout: key = AQCYmgZmuFP2BBAAoKiOEFakmPWdysHHeXfdNQ== 2024-03-29T10:40:33.660 INFO:teuthology.orchestra.run.smithi088.stdout:[osd.6] 2024-03-29T10:40:33.660 INFO:teuthology.orchestra.run.smithi088.stdout: key = AQCbmgZmcN9iAhAAqQZySUrPTFrPDrzryi79Nw== 2024-03-29T10:40:33.661 INFO:teuthology.orchestra.run.smithi088.stdout:[osd.7] 2024-03-29T10:40:33.661 INFO:teuthology.orchestra.run.smithi088.stdout: key = AQCemgZm7xOIABAAKAi2/yuYHP+KWRx8iAnGhg== 2024-03-29T10:40:33.661 INFO:teuthology.orchestra.run.smithi088.stdout:[client.0] 2024-03-29T10:40:33.661 INFO:teuthology.orchestra.run.smithi088.stdout: key = AQCCmgZmvS3lKRAAsWhvpE52ytBlkYdnAe0PPw== 2024-03-29T10:40:33.661 INFO:teuthology.orchestra.run.smithi088.stdout:[client.1] 2024-03-29T10:40:33.661 INFO:teuthology.orchestra.run.smithi088.stdout: key = AQCCmgZmxhZ3LRAABxNQXKik0g8pZnpMKP04OQ== 2024-03-29T10:40:33.677 INFO:teuthology.orchestra.run.smithi196.stdout:[mgr.y] 2024-03-29T10:40:33.677 INFO:teuthology.orchestra.run.smithi196.stdout: key = AQCCmgZmTrwWIxAAJClcgAW5xlcXBD3cjHZw/Q== 2024-03-29T10:40:33.677 INFO:teuthology.orchestra.run.smithi196.stdout:[osd.0] 2024-03-29T10:40:33.678 INFO:teuthology.orchestra.run.smithi196.stdout: key = AQCGmgZmfaHNABAAET1tUwl7FhyzxFwteB2oYw== 2024-03-29T10:40:33.678 INFO:teuthology.orchestra.run.smithi196.stdout:[osd.1] 2024-03-29T10:40:33.678 INFO:teuthology.orchestra.run.smithi196.stdout: key = AQCImgZmR1chOxAAcJRY+ZwP3JAzG+SOC+hDKQ== 2024-03-29T10:40:33.678 INFO:teuthology.orchestra.run.smithi196.stdout:[osd.2] 2024-03-29T10:40:33.678 INFO:teuthology.orchestra.run.smithi196.stdout: key = AQCLmgZmrV6FORAArl5tDJX0kIDz3KG43VfQpw== 2024-03-29T10:40:33.678 INFO:teuthology.orchestra.run.smithi196.stdout:[osd.3] 2024-03-29T10:40:33.678 INFO:teuthology.orchestra.run.smithi196.stdout: key = AQCOmgZm9mXANxAAldvBkyxpNEJCOYlOGKLJEg== 2024-03-29T10:40:33.678 INFO:teuthology.orchestra.run.smithi196.stdout:[mgr.x] 2024-03-29T10:40:33.678 INFO:teuthology.orchestra.run.smithi196.stdout: key = AQCCmgZm3eIRJxAAV+wluWTgoxHkFAneC1IJ8w== 2024-03-29T10:40:33.678 INFO:teuthology.orchestra.run.smithi196.stdout:[osd.4] 2024-03-29T10:40:33.679 INFO:teuthology.orchestra.run.smithi196.stdout: key = AQCVmgZmq4rCBhAAvAx2dbSZZyxglekYKq/85g== 2024-03-29T10:40:33.679 INFO:teuthology.orchestra.run.smithi196.stdout:[osd.5] 2024-03-29T10:40:33.679 INFO:teuthology.orchestra.run.smithi196.stdout: key = AQCYmgZmuFP2BBAAoKiOEFakmPWdysHHeXfdNQ== 2024-03-29T10:40:33.679 INFO:teuthology.orchestra.run.smithi196.stdout:[osd.6] 2024-03-29T10:40:33.679 INFO:teuthology.orchestra.run.smithi196.stdout: key = AQCbmgZmcN9iAhAAqQZySUrPTFrPDrzryi79Nw== 2024-03-29T10:40:33.679 INFO:teuthology.orchestra.run.smithi196.stdout:[osd.7] 2024-03-29T10:40:33.679 INFO:teuthology.orchestra.run.smithi196.stdout: key = AQCemgZm7xOIABAAKAi2/yuYHP+KWRx8iAnGhg== 2024-03-29T10:40:33.679 INFO:teuthology.orchestra.run.smithi196.stdout:[client.0] 2024-03-29T10:40:33.679 INFO:teuthology.orchestra.run.smithi196.stdout: key = AQCCmgZmvS3lKRAAsWhvpE52ytBlkYdnAe0PPw== 2024-03-29T10:40:33.680 INFO:teuthology.orchestra.run.smithi196.stdout:[client.1] 2024-03-29T10:40:33.680 INFO:teuthology.orchestra.run.smithi196.stdout: key = AQCCmgZmxhZ3LRAABxNQXKik0g8pZnpMKP04OQ== 2024-03-29T10:40:33.681 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-29T10:40:33.711 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-29T10:40:33.775 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:40:33.779 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:40:33.835 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:40:33.837 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:40:33.892 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:40:33.897 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:40:33.952 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:40:33.956 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:40:34.012 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-29T10:40:34.015 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-29T10:40:34.070 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:40:34.076 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:40:34.130 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:40:34.132 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:40:34.184 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:40:34.187 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:40:34.240 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:40:34.243 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:40:34.294 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-29T10:40:34.296 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-29T10:40:34.351 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-29T10:40:34.354 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-29T10:40:34.409 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-03-29T10:40:34.409 DEBUG:teuthology.orchestra.run.smithi088:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-03-29T10:40:34.435 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-29T10:40:34.532 DEBUG:teuthology.orchestra.run.smithi088:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-03-29T10:40:34.559 DEBUG:teuthology.orchestra.run.smithi088:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-03-29T10:40:34.627 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-29T10:40:34.723 DEBUG:teuthology.orchestra.run.smithi088:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-03-29T10:40:34.750 DEBUG:teuthology.orchestra.run.smithi196:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-03-29T10:40:34.781 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-29T10:40:34.879 DEBUG:teuthology.orchestra.run.smithi196:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-03-29T10:40:34.905 DEBUG:teuthology.orchestra.run.smithi088:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-29T10:40:34.908 DEBUG:teuthology.orchestra.run.smithi196:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-29T10:40:34.961 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-03-29T10:40:34.961 INFO:tasks.ceph.mon.a:Restarting daemon 2024-03-29T10:40:34.961 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-03-29T10:40:34.964 INFO:tasks.ceph.mon.a:Started 2024-03-29T10:40:34.964 INFO:tasks.ceph.mon.c:Restarting daemon 2024-03-29T10:40:34.964 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-03-29T10:40:34.966 INFO:tasks.ceph.mon.c:Started 2024-03-29T10:40:34.966 INFO:tasks.ceph.mon.b:Restarting daemon 2024-03-29T10:40:34.966 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-03-29T10:40:35.003 INFO:tasks.ceph.mon.b:Started 2024-03-29T10:40:35.003 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-03-29T10:40:35.003 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-03-29T10:40:35.003 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2024-03-29T10:40:35.005 INFO:tasks.ceph.mgr.y:Started 2024-03-29T10:40:35.005 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-03-29T10:40:35.006 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-03-29T10:40:35.008 INFO:tasks.ceph.mgr.x:Started 2024-03-29T10:40:35.008 INFO:tasks.ceph:Setting crush tunables to default 2024-03-29T10:40:35.008 DEBUG:teuthology.orchestra.run.smithi088:> sudo ceph --cluster ceph osd crush tunables default 2024-03-29T10:40:35.109 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:35.125+0000 7fbcfc01e640 10 monclient: get_monmap_and_config 2024-03-29T10:40:35.109 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:35.125+0000 7fbcfc01e640 10 monclient: build_initial_monmap 2024-03-29T10:40:35.109 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:35.125+0000 7fbcfc01e640 10 monclient: monmap: 2024-03-29T10:40:35.109 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2024-03-29T10:40:35.109 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:40:35.109 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-03-29T10:40:35.126408+0000 2024-03-29T10:40:35.109 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-03-29T10:40:35.126408+0000 2024-03-29T10:40:35.109 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2024-03-29T10:40:35.109 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2024-03-29T10:40:35.109 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-03-29T10:40:35.110 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-29T10:40:35.110 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-b 2024-03-29T10:40:35.110 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-03-29T10:40:35.111 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:35.125+0000 7fbcfc01e640 1 Processor -- start 2024-03-29T10:40:35.111 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:35.126+0000 7fbcfc01e640 1 -- start start 2024-03-29T10:40:35.111 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:35.126+0000 7fbcfc01e640 10 monclient: init 2024-03-29T10:40:35.111 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:35.126+0000 7fbcfc01e640 10 monclient: _reopen_session rank -1 2024-03-29T10:40:35.112 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:35.126+0000 7fbcfc01e640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:40:35.112 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:35.126+0000 7fbcfc01e640 10 monclient(hunting): picked mon.noname-b con 0x7fbcf40b8240 addr v1:172.21.15.88:6790/0 2024-03-29T10:40:35.112 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:35.126+0000 7fbcfc01e640 10 monclient(hunting): picked mon.noname-a con 0x7fbcf40b1f60 addr v1:172.21.15.88:6789/0 2024-03-29T10:40:35.112 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:35.126+0000 7fbcfc01e640 10 monclient(hunting): picked mon.noname-c con 0x7fbcf409c8c0 addr v1:172.21.15.196:6789/0 2024-03-29T10:40:35.112 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:35.126+0000 7fbcfc01e640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbcf409bd80 con 0x7fbcf40b1f60 2024-03-29T10:40:35.112 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:35.126+0000 7fbcfc01e640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbcf409b960 con 0x7fbcf409c8c0 2024-03-29T10:40:35.112 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:35.126+0000 7fbcfc01e640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbcf40b2ec0 con 0x7fbcf40b8240 2024-03-29T10:40:35.112 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:35.126+0000 7fbcfc01e640 10 monclient(hunting): _renew_subs 2024-03-29T10:40:35.112 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:35.126+0000 7fbcfc01e640 10 monclient(hunting): authenticate will time out at 757.362617s 2024-03-29T10:40:35.112 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:35.126+0000 7fbcf9dba640 1 --1- >> v1:172.21.15.88:6790/0 conn(0x7fbcf40b8240 0x7fbcf413f450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6790/0 says I am v1:172.21.15.88:34052/0 (socket says 172.21.15.88:34052) 2024-03-29T10:40:35.112 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:35.126+0000 7fbcf95b9640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fbcf40b1f60 0x7fbcf409c1b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:44698/0 (socket says 172.21.15.88:44698) 2024-03-29T10:40:35.112 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:35.126+0000 7fbcf9dba640 1 -- 172.21.15.88:0/129118169 learned_addr learned my addr 172.21.15.88:0/129118169 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-29T10:40:38.111 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:38.126+0000 7fbceb7fe640 10 monclient(hunting): tick 2024-03-29T10:40:38.111 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:38.126+0000 7fbceb7fe640 10 monclient(hunting): _check_auth_tickets 2024-03-29T10:40:38.111 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:38.126+0000 7fbceb7fe640 1 monclient(hunting): continuing hunt 2024-03-29T10:40:38.111 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:38.126+0000 7fbceb7fe640 10 monclient(hunting): _reopen_session rank -1 2024-03-29T10:40:38.111 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:38.126+0000 7fbceb7fe640 1 -- 172.21.15.88:0/129118169 >> v1:172.21.15.88:6790/0 conn(0x7fbcf40b8240 legacy=0x7fbcf413f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:38.111 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:38.126+0000 7fbceb7fe640 1 -- 172.21.15.88:0/129118169 >> v1:172.21.15.196:6789/0 conn(0x7fbcf409c8c0 legacy=0x7fbcf414d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:38.111 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:38.126+0000 7fbceb7fe640 1 -- 172.21.15.88:0/129118169 >> v1:172.21.15.88:6789/0 conn(0x7fbcf40b1f60 legacy=0x7fbcf409c1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:38.111 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:38.126+0000 7fbceb7fe640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:40:38.111 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:38.126+0000 7fbceb7fe640 10 monclient(hunting): picked mon.noname-c con 0x7fbcd80036e0 addr v1:172.21.15.196:6789/0 2024-03-29T10:40:38.111 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:38.126+0000 7fbceb7fe640 10 monclient(hunting): picked mon.noname-a con 0x7fbcd80072d0 addr v1:172.21.15.88:6789/0 2024-03-29T10:40:38.111 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:38.126+0000 7fbceb7fe640 10 monclient(hunting): picked mon.noname-b con 0x7fbcf40b8240 addr v1:172.21.15.88:6790/0 2024-03-29T10:40:38.112 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:38.126+0000 7fbceb7fe640 1 -- 172.21.15.88:0/129118169 --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbcf413fb60 con 0x7fbcd80072d0 2024-03-29T10:40:38.112 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:38.126+0000 7fbceb7fe640 1 -- 172.21.15.88:0/129118169 --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbcd800f8d0 con 0x7fbcd80036e0 2024-03-29T10:40:38.112 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:38.126+0000 7fbceb7fe640 1 -- 172.21.15.88:0/129118169 --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbcd8010b10 con 0x7fbcf40b8240 2024-03-29T10:40:38.112 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:38.126+0000 7fbceb7fe640 10 monclient(hunting): _renew_subs 2024-03-29T10:40:40.286 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:40.300+0000 7fbcf8db8640 1 -- 172.21.15.88:0/129118169 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3822745327 0 0) 0x7fbcf413fb60 con 0x7fbcd80072d0 2024-03-29T10:40:40.286 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:40.300+0000 7fbcf8db8640 10 monclient(hunting): _negotiate 2024-03-29T10:40:40.286 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:40.300+0000 7fbcf8db8640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:40.286 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:40.300+0000 7fbcf8db8640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:40.286 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:40.300+0000 7fbcf8db8640 10 monclient(hunting): my global_id is 4101 2024-03-29T10:40:40.286 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:40.300+0000 7fbcf8db8640 1 -- 172.21.15.88:0/129118169 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbce0003520 con 0x7fbcd80072d0 2024-03-29T10:40:40.287 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:40.301+0000 7fbcf8db8640 1 -- 172.21.15.88:0/129118169 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3267055311 0 0) 0x7fbcd800f8d0 con 0x7fbcd80036e0 2024-03-29T10:40:40.287 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:40.301+0000 7fbcf8db8640 10 monclient(hunting): _negotiate 2024-03-29T10:40:40.287 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:40.301+0000 7fbcf8db8640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:40.287 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:40.301+0000 7fbcf8db8640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:40.287 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:40.301+0000 7fbcf8db8640 10 monclient(hunting): my global_id is 4105 2024-03-29T10:40:40.287 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:40.301+0000 7fbcf8db8640 1 -- 172.21.15.88:0/129118169 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbcf413fb60 con 0x7fbcd80036e0 2024-03-29T10:40:40.287 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:40.301+0000 7fbcf8db8640 1 -- 172.21.15.88:0/129118169 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2151102297 0 0) 0x7fbcf413fb60 con 0x7fbcd80036e0 2024-03-29T10:40:40.287 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:40.301+0000 7fbcf8db8640 1 -- 172.21.15.88:0/129118169 >> v1:172.21.15.88:6790/0 conn(0x7fbcf40b8240 legacy=0x7fbcd800d070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:40.287 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:40.301+0000 7fbcf8db8640 1 -- 172.21.15.88:0/129118169 >> v1:172.21.15.88:6789/0 conn(0x7fbcd80072d0 legacy=0x7fbcd8009760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:40.287 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:40.301+0000 7fbcf8db8640 10 monclient: _finish_hunting 0 2024-03-29T10:40:40.287 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:40.301+0000 7fbcf8db8640 1 monclient: found mon.noname-c 2024-03-29T10:40:40.287 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:40.301+0000 7fbcf8db8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:40:40.287 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:40.301+0000 7fbcf8db8640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.196:6789/0 2024-03-29T10:40:40.288 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:40.301+0000 7fbcf8db8640 1 -- 172.21.15.88:0/129118169 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcd8011d70 con 0x7fbcd80036e0 2024-03-29T10:40:40.288 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:40.301+0000 7fbcf8db8640 10 monclient: _finish_auth 0 2024-03-29T10:40:40.288 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:40.301+0000 7fbcf8db8640 10 monclient: _check_auth_tickets 2024-03-29T10:40:40.288 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:40.301+0000 7fbcf8db8640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:40:40.288 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:40.301+0000 7fbcf8db8640 1 -- 172.21.15.88:0/129118169 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7fbcf0001690 con 0x7fbcd80036e0 2024-03-29T10:40:40.288 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:40.301+0000 7fbcf8db8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:40.288 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:40.301+0000 7fbcf8db8640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:40:40.288 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:40.301+0000 7fbcf8db8640 10 monclient: dump: 2024-03-29T10:40:40.288 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-03-29T10:40:40.288 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:40:40.288 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:40:40.288 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:40:40.288 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:40.288 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2024-03-29T10:40:40.289 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:40:40.289 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:40:40.289 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:40:40.289 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-03-29T10:40:40.289 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:40.301+0000 7fbcfc01e640 5 monclient: authenticate success, global_id 4105 2024-03-29T10:40:40.289 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:40.301+0000 7fbcfc01e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:40:40.289 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:40.301+0000 7fbcf8db8640 1 -- 172.21.15.88:0/129118169 <== mon.1 v1:172.21.15.196:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbcf0001860 con 0x7fbcd80036e0 2024-03-29T10:40:40.289 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:40.301+0000 7fbcf8db8640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:40:40.289 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:40.301+0000 7fbcf8db8640 1 -- 172.21.15.88:0/129118169 <== mon.1 v1:172.21.15.196:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7fbcf0001bf0 con 0x7fbcd80036e0 2024-03-29T10:40:40.289 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:40.301+0000 7fbcf8db8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:40.289 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:40.301+0000 7fbcf8db8640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:40:40.289 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:40.301+0000 7fbcf8db8640 10 monclient: dump: 2024-03-29T10:40:40.289 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-03-29T10:40:40.289 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:40:40.290 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:40:40.290 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:40:40.290 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:40.290 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2024-03-29T10:40:40.290 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:40:40.290 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:40:40.290 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:40:40.290 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-03-29T10:40:40.290 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:40.301+0000 7fbcfc01e640 10 monclient: get_monmap_and_config success 2024-03-29T10:40:40.291 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:40.301+0000 7fbcfc01e640 4 set_mon_vals no callback set 2024-03-29T10:40:40.291 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:40.302+0000 7fbcfc01e640 10 monclient: shutdown 2024-03-29T10:40:40.291 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:40.302+0000 7fbcfc01e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:40:40.291 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:40.302+0000 7fbcfc01e640 1 -- 172.21.15.88:0/129118169 >> v1:172.21.15.196:6789/0 conn(0x7fbcd80036e0 legacy=0x7fbcd8005b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:40.291 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:40.302+0000 7fbcfc01e640 1 -- 172.21.15.88:0/129118169 shutdown_connections 2024-03-29T10:40:40.291 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:40.302+0000 7fbcfc01e640 1 -- 172.21.15.88:0/129118169 >> 172.21.15.88:0/129118169 conn(0x7fbcf4066b00 msgr2=0x7fbcf4066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:40:40.291 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:40.302+0000 7fbcfc01e640 1 -- 172.21.15.88:0/129118169 shutdown_connections 2024-03-29T10:40:40.291 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:40.302+0000 7fbcfc01e640 1 -- 172.21.15.88:0/129118169 wait complete. 2024-03-29T10:40:40.295 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:40.303+0000 7fbcfc01e640 10 monclient: build_initial_monmap 2024-03-29T10:40:40.295 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:40.303+0000 7fbcfc01e640 10 monclient: monmap: 2024-03-29T10:40:40.295 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2024-03-29T10:40:40.296 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:40:40.296 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 0.000000 2024-03-29T10:40:40.296 INFO:teuthology.orchestra.run.smithi088.stderr:created 0.000000 2024-03-29T10:40:40.296 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2024-03-29T10:40:40.296 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2024-03-29T10:40:40.296 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-03-29T10:40:40.296 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-29T10:40:40.296 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-c 2024-03-29T10:40:40.296 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-03-29T10:40:40.296 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:40.304+0000 7fbcfc01e640 1 librados: starting msgr at 2024-03-29T10:40:40.296 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:40.304+0000 7fbcfc01e640 1 librados: starting objecter 2024-03-29T10:40:40.297 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:40.304+0000 7fbcfc01e640 1 Processor -- start 2024-03-29T10:40:40.297 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:40.304+0000 7fbcfc01e640 1 -- start start 2024-03-29T10:40:40.297 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:40.304+0000 7fbcfc01e640 1 librados: setting wanted keys 2024-03-29T10:40:40.297 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:40.304+0000 7fbcfc01e640 1 librados: calling monclient init 2024-03-29T10:40:40.297 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:40.304+0000 7fbcfc01e640 10 monclient: init 2024-03-29T10:40:40.297 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:40.305+0000 7fbcfc01e640 10 monclient: _reopen_session rank -1 2024-03-29T10:40:40.297 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:40.305+0000 7fbcfc01e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:40:40.297 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:40.305+0000 7fbcfc01e640 10 monclient(hunting): picked mon.noname-a con 0x7fbcd80036e0 addr v1:172.21.15.88:6789/0 2024-03-29T10:40:40.297 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:40.305+0000 7fbcfc01e640 10 monclient(hunting): picked mon.noname-c con 0x7fbcd80072d0 addr v1:172.21.15.88:6790/0 2024-03-29T10:40:40.297 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:40.305+0000 7fbcfc01e640 10 monclient(hunting): picked mon.noname-b con 0x7fbcf40b8240 addr v1:172.21.15.196:6789/0 2024-03-29T10:40:40.297 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:40.305+0000 7fbcfc01e640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbcf40b17d0 con 0x7fbcd80036e0 2024-03-29T10:40:40.297 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:40.305+0000 7fbcfc01e640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbcf40b0e80 con 0x7fbcf40b8240 2024-03-29T10:40:40.297 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:40.305+0000 7fbcfc01e640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbcf40b1a40 con 0x7fbcd80072d0 2024-03-29T10:40:40.298 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:40.305+0000 7fbcfc01e640 10 monclient(hunting): _renew_subs 2024-03-29T10:40:40.298 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:40.305+0000 7fbcfc01e640 10 monclient(hunting): authenticate will time out at 762.541324s 2024-03-29T10:40:40.298 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:40.305+0000 7fbcf95b9640 1 --1- >> v1:172.21.15.88:6790/0 conn(0x7fbcd80072d0 0x7fbcf4138da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6790/0 says I am v1:172.21.15.88:34064/0 (socket says 172.21.15.88:34064) 2024-03-29T10:40:40.298 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:40.305+0000 7fbcf9dba640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fbcd80036e0 0x7fbcf40b0770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:44730/0 (socket says 172.21.15.88:44730) 2024-03-29T10:40:40.298 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:40.305+0000 7fbcf95b9640 1 -- 172.21.15.88:0/1587218627 learned_addr learned my addr 172.21.15.88:0/1587218627 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-29T10:40:40.298 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:40.305+0000 7fbcea7fc640 10 client.?.objecter ms_handle_connect 0x7fbcd80036e0 2024-03-29T10:40:40.298 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:40.305+0000 7fbcea7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:40:40.298 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:40.305+0000 7fbcea7fc640 10 client.?.objecter ms_handle_connect 0x7fbcf40b8240 2024-03-29T10:40:40.298 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:40.305+0000 7fbcea7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:40:40.298 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:40.305+0000 7fbcea7fc640 10 client.?.objecter ms_handle_connect 0x7fbcd80072d0 2024-03-29T10:40:40.298 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:40.305+0000 7fbcea7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:40:40.298 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:40.305+0000 7fbcea7fc640 1 -- 172.21.15.88:0/1587218627 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3000502346 0 0) 0x7fbcf40b1a40 con 0x7fbcd80072d0 2024-03-29T10:40:40.298 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:40.305+0000 7fbcea7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:40:40.298 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:40.305+0000 7fbcea7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:40.299 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:40.305+0000 7fbcea7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:40.299 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:40.305+0000 7fbcea7fc640 10 monclient(hunting): my global_id is 4103 2024-03-29T10:40:40.299 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:40.305+0000 7fbcea7fc640 1 -- 172.21.15.88:0/1587218627 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbcd8011570 con 0x7fbcd80072d0 2024-03-29T10:40:40.299 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:40.306+0000 7fbcfa5bb640 1 -- 172.21.15.88:0/1587218627 >> v1:172.21.15.196:6789/0 conn(0x7fbcf40b8240 legacy=0x7fbcf40ad3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-03-29T10:40:40.299 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:40.306+0000 7fbcfa5bb640 1 -- 172.21.15.88:0/1587218627 >> v1:172.21.15.196:6789/0 conn(0x7fbcf40b8240 legacy=0x7fbcf40ad3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-29T10:40:40.299 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:40.306+0000 7fbcfa5bb640 1 --1- 172.21.15.88:0/1587218627 >> v1:172.21.15.196:6789/0 conn(0x7fbcf40b8240 0x7fbcf40ad3e0 :-1 s=OPENED pgs=20 cs=1 l=1).handle_message read tag failed 2024-03-29T10:40:40.299 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:40.306+0000 7fbcfa5bb640 1 --1- 172.21.15.88:0/1587218627 >> v1:172.21.15.196:6789/0 conn(0x7fbcf40b8240 0x7fbcf40ad3e0 :-1 s=OPENED pgs=20 cs=1 l=1).fault on lossy channel, failing 2024-03-29T10:40:40.299 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:40.306+0000 7fbcea7fc640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.196:6789/0 2024-03-29T10:40:40.299 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:40.306+0000 7fbcea7fc640 1 -- 172.21.15.88:0/1587218627 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1010107710 0 0) 0x7fbcd8011570 con 0x7fbcd80072d0 2024-03-29T10:40:40.299 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:40.306+0000 7fbcea7fc640 1 -- 172.21.15.88:0/1587218627 --> v1:172.21.15.88:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbcf40b1a40 con 0x7fbcd80072d0 2024-03-29T10:40:40.299 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:40.306+0000 7fbcea7fc640 1 -- 172.21.15.88:0/1587218627 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7fbce4001820 con 0x7fbcd80072d0 2024-03-29T10:40:40.299 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:40.306+0000 7fbcea7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:40.299 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:40.306+0000 7fbcea7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:40:40.299 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:40.306+0000 7fbcea7fc640 10 monclient(hunting): dump: 2024-03-29T10:40:40.300 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-03-29T10:40:40.300 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:40:40.300 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:40:40.300 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:40:40.300 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:40.300 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2024-03-29T10:40:40.300 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:40:40.300 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:40:40.300 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:40:40.300 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-03-29T10:40:40.300 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:40.306+0000 7fbcea7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:40:40.300 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:40.306+0000 7fbcea7fc640 1 -- 172.21.15.88:0/1587218627 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3222398658 0 0) 0x7fbcf40b17d0 con 0x7fbcd80036e0 2024-03-29T10:40:40.300 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:40.306+0000 7fbcea7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:40:40.300 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:40.306+0000 7fbcea7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:40.300 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:40.306+0000 7fbcea7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:40.301 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:40.306+0000 7fbcea7fc640 10 monclient(hunting): my global_id is 4107 2024-03-29T10:40:40.301 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:40.306+0000 7fbcea7fc640 1 -- 172.21.15.88:0/1587218627 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbcd8011570 con 0x7fbcd80036e0 2024-03-29T10:40:40.301 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:40.306+0000 7fbcea7fc640 1 -- 172.21.15.88:0/1587218627 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2440207465 0 0) 0x7fbcd8011570 con 0x7fbcd80036e0 2024-03-29T10:40:40.301 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:40.306+0000 7fbcea7fc640 1 -- 172.21.15.88:0/1587218627 --> v1:172.21.15.88:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbcf40b17d0 con 0x7fbcd80036e0 2024-03-29T10:40:40.301 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:40.306+0000 7fbcea7fc640 1 -- 172.21.15.88:0/1587218627 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7fbcf0001b70 con 0x7fbcd80036e0 2024-03-29T10:40:40.301 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:40.306+0000 7fbcea7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:40.301 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:40.306+0000 7fbcea7fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:40:40.301 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:40.306+0000 7fbcea7fc640 10 monclient(hunting): dump: 2024-03-29T10:40:40.301 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-03-29T10:40:40.301 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:40:40.301 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:40:40.301 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:40:40.301 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:40.301 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2024-03-29T10:40:40.301 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:40:40.302 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:40:40.302 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:40:40.302 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-03-29T10:40:40.302 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:40.307+0000 7fbcea7fc640 1 -- 172.21.15.88:0/1587218627 <== mon.0 v1:172.21.15.88:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1645530028 0 0) 0x7fbcf40b17d0 con 0x7fbcd80036e0 2024-03-29T10:40:40.302 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:40.307+0000 7fbcea7fc640 1 -- 172.21.15.88:0/1587218627 >> v1:172.21.15.88:6790/0 conn(0x7fbcd80072d0 legacy=0x7fbcf4138da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:40.302 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:40.307+0000 7fbcea7fc640 1 -- 172.21.15.88:0/1587218627 >> v1:172.21.15.196:6789/0 conn(0x7fbcf40b8240 legacy=0x7fbcf40ad3e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-29T10:40:40.302 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:40.307+0000 7fbcea7fc640 10 monclient: _finish_hunting 0 2024-03-29T10:40:40.302 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:40.307+0000 7fbcea7fc640 1 monclient: found mon.a 2024-03-29T10:40:40.302 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:40.307+0000 7fbcea7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:40:40.302 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:40.307+0000 7fbcea7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-03-29T10:40:40.302 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:40.307+0000 7fbcea7fc640 1 -- 172.21.15.88:0/1587218627 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcf40afe30 con 0x7fbcd80036e0 2024-03-29T10:40:40.302 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:40.307+0000 7fbcea7fc640 10 monclient: _finish_auth 0 2024-03-29T10:40:40.302 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:40.307+0000 7fbcea7fc640 10 monclient: _check_auth_tickets 2024-03-29T10:40:40.302 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:40.307+0000 7fbcea7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:40:40.303 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:40.307+0000 7fbcfc01e640 5 monclient: authenticate success, global_id 4107 2024-03-29T10:40:40.303 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:40.307+0000 7fbcfc01e640 10 monclient: _renew_subs 2024-03-29T10:40:40.303 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:40.307+0000 7fbcfc01e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-03-29T10:40:40.303 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:40.307+0000 7fbcfc01e640 1 -- 172.21.15.88:0/1587218627 --> v1:172.21.15.88:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbcf40af9a0 con 0x7fbcd80036e0 2024-03-29T10:40:40.303 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:40.307+0000 7fbcfc01e640 10 client.4107.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:40:40.303 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:40.307+0000 7fbcfc01e640 10 monclient: _renew_subs 2024-03-29T10:40:40.303 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:40.307+0000 7fbcfc01e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-03-29T10:40:40.303 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:40.307+0000 7fbcfc01e640 1 -- 172.21.15.88:0/1587218627 --> v1:172.21.15.88:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbcf4139690 con 0x7fbcd80036e0 2024-03-29T10:40:40.303 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:40.307+0000 7fbcfc01e640 1 librados: init done 2024-03-29T10:40:40.303 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:40.307+0000 7fbcea7fc640 1 -- 172.21.15.88:0/1587218627 <== mon.0 v1:172.21.15.88:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbcf0001070 con 0x7fbcd80036e0 2024-03-29T10:40:40.303 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:40.307+0000 7fbcea7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:40:40.303 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:40.307+0000 7fbcea7fc640 1 -- 172.21.15.88:0/1587218627 <== mon.0 v1:172.21.15.88:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7fbcf0001d50 con 0x7fbcd80036e0 2024-03-29T10:40:40.303 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:40.307+0000 7fbcea7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:40.303 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:40.307+0000 7fbcea7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:40:40.304 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:40.307+0000 7fbcebfff640 4 set_mon_vals no callback set 2024-03-29T10:40:40.304 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:40.307+0000 7fbcea7fc640 10 monclient: dump: 2024-03-29T10:40:40.304 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-03-29T10:40:40.304 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:40:40.304 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:40:40.304 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:40:40.304 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:40.304 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2024-03-29T10:40:40.304 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:40:40.304 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:40:40.304 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:40:40.304 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-03-29T10:40:40.304 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:40.307+0000 7fbcea7fc640 1 -- 172.21.15.88:0/1587218627 <== mon.0 v1:172.21.15.88:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fbcf0005080 con 0x7fbcd80036e0 2024-03-29T10:40:40.304 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:40.307+0000 7fbcea7fc640 1 -- 172.21.15.88:0/1587218627 <== mon.0 v1:172.21.15.88:6789/0 8 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (unknown 926703052 0 0) 0x7fbcf00065f0 con 0x7fbcd80036e0 2024-03-29T10:40:40.304 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:40.307+0000 7fbcea7fc640 10 client.4107.objecter ms_dispatch 0x7fbcf4000b90 osd_map(1..1 src has 1..1) v4 2024-03-29T10:40:40.305 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:40.307+0000 7fbcea7fc640 3 client.4107.objecter handle_osd_map got epochs [1,1] > 0 2024-03-29T10:40:40.305 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:40.307+0000 7fbcea7fc640 3 client.4107.objecter handle_osd_map decoding full epoch 1 2024-03-29T10:40:40.305 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:40.307+0000 7fbcea7fc640 20 client.4107.objecter dump_active .. 0 homeless 2024-03-29T10:40:40.305 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:40.308+0000 7fbcfc01e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:40:40.305 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:40.308+0000 7fbcfc01e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:40:40.305 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:40.308+0000 7fbcfc01e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-03-29T10:40:40.305 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:40.308+0000 7fbcfc01e640 1 -- 172.21.15.88:0/1587218627 --> v1:172.21.15.88:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbcdc005080 con 0x7fbcd80036e0 2024-03-29T10:40:40.305 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:40.312+0000 7fbcea7fc640 1 -- 172.21.15.88:0/1587218627 <== mon.0 v1:172.21.15.88:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fbcf0006800 con 0x7fbcd80036e0 2024-03-29T10:40:40.305 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:40.312+0000 7fbcea7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:40:40.305 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:40.312+0000 7fbcea7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:40:40.391 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:40.408+0000 7fbcfc01e640 10 start_mon_command cmd=[{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-29T10:40:40.392 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:40.408+0000 7fbcfc01e640 10 monclient: _send_command 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-29T10:40:40.392 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:40.408+0000 7fbcfc01e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-03-29T10:40:40.392 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:40.408+0000 7fbcfc01e640 1 -- 172.21.15.88:0/1587218627 --> v1:172.21.15.88:6789/0 -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7fbcdc005330 con 0x7fbcd80036e0 2024-03-29T10:40:40.393 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:40.409+0000 7fbcea7fc640 1 -- 172.21.15.88:0/1587218627 <== mon.0 v1:172.21.15.88:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (unknown 2277712955 0 0) 0x7fbcf0005630 con 0x7fbcd80036e0 2024-03-29T10:40:40.393 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:40.409+0000 7fbcea7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-29T10:40:40.393 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:40.409+0000 7fbcea7fc640 10 monclient: _finish_command 2 = system:0 adjusted tunables profile to default 2024-03-29T10:40:40.393 INFO:teuthology.orchestra.run.smithi088.stderr:adjusted tunables profile to default 2024-03-29T10:40:40.395 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:40.410+0000 7fbcfc01e640 10 librados: watch_flush enter 2024-03-29T10:40:40.395 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:40.410+0000 7fbcfc01e640 10 librados: watch_flush exit 2024-03-29T10:40:40.395 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:40.410+0000 7fbcfc01e640 20 client.4107.objecter shutdown clearing up homeless session... 2024-03-29T10:40:40.395 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:40.410+0000 7fbcfc01e640 10 client.4107.objecter successfully canceled tick 2024-03-29T10:40:40.395 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:40.410+0000 7fbcfc01e640 10 monclient: shutdown 2024-03-29T10:40:40.395 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:40.410+0000 7fbcfc01e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:40:40.395 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:40.410+0000 7fbcfc01e640 1 -- 172.21.15.88:0/1587218627 >> v1:172.21.15.88:6789/0 conn(0x7fbcd80036e0 legacy=0x7fbcf40b0770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:40.395 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:40.410+0000 7fbcfc01e640 1 -- 172.21.15.88:0/1587218627 shutdown_connections 2024-03-29T10:40:40.395 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:40.410+0000 7fbcfc01e640 1 -- 172.21.15.88:0/1587218627 >> 172.21.15.88:0/1587218627 conn(0x7fbcf4066b00 msgr2=0x7fbcf413e1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:40:40.395 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:40.411+0000 7fbcfc01e640 1 -- 172.21.15.88:0/1587218627 shutdown_connections 2024-03-29T10:40:40.396 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:40.411+0000 7fbcfc01e640 1 -- 172.21.15.88:0/1587218627 wait complete. 2024-03-29T10:40:40.396 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:40.411+0000 7fbcfc01e640 1 librados: shutdown 2024-03-29T10:40:40.405 INFO:tasks.ceph:check_enable_crimson: False 2024-03-29T10:40:40.405 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-03-29T10:40:40.406 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-03-29T10:40:40.406 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-03-29T10:40:40.432 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-03-29T10:40:40.432 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-03-29T10:40:40.499 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-03-29T10:40:40.499 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-03-29T10:40:40.567 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-03-29T10:40:40.567 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-03-29T10:40:40.634 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2024-03-29T10:40:40.634 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-03-29T10:40:40.662 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2024-03-29T10:40:40.662 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-03-29T10:40:40.728 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2024-03-29T10:40:40.728 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-03-29T10:40:40.762 INFO:tasks.ceph.mgr.x.smithi196.stderr:2024-03-29T10:40:40.778+0000 7ff5dcc4b040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-29T10:40:40.787 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2024-03-29T10:40:40.788 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-03-29T10:40:40.813 INFO:tasks.ceph.mgr.y.smithi088.stderr:2024-03-29T10:40:40.829+0000 7fca56440040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-29T10:40:40.817 INFO:tasks.ceph.mgr.x.smithi196.stderr:2024-03-29T10:40:40.834+0000 7ff5dcc4b040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-29T10:40:40.843 DEBUG:teuthology.orchestra.run.smithi196:> sudo ceph --cluster ceph osd new a26c0c1c-8de0-438c-a3f6-3df10ea61574 0 2024-03-29T10:40:40.867 INFO:tasks.ceph.mgr.y.smithi088.stderr:2024-03-29T10:40:40.884+0000 7fca56440040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-29T10:40:40.984 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:40.999+0000 7f5f01a43640 10 monclient: get_monmap_and_config 2024-03-29T10:40:40.984 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.000+0000 7f5f01a43640 10 monclient: build_initial_monmap 2024-03-29T10:40:40.984 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.000+0000 7f5f01a43640 10 monclient: monmap: 2024-03-29T10:40:40.984 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 0 2024-03-29T10:40:40.984 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:40:40.984 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 2024-03-29T10:40:41.000890+0000 2024-03-29T10:40:40.984 INFO:teuthology.orchestra.run.smithi196.stderr:created 2024-03-29T10:40:41.000890+0000 2024-03-29T10:40:40.984 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 0 (unknown) 2024-03-29T10:40:40.984 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 1 2024-03-29T10:40:40.984 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-03-29T10:40:40.985 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-29T10:40:40.985 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.88:6790/0 mon.noname-b 2024-03-29T10:40:40.985 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-29T10:40:40.985 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.000+0000 7f5f01a43640 1 Processor -- start 2024-03-29T10:40:40.985 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.000+0000 7f5f01a43640 1 -- start start 2024-03-29T10:40:40.985 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.000+0000 7f5f01a43640 10 monclient: init 2024-03-29T10:40:40.993 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.001+0000 7f5f01a43640 10 monclient: _reopen_session rank -1 2024-03-29T10:40:40.994 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.001+0000 7f5f01a43640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:40:40.994 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.001+0000 7f5f01a43640 10 monclient(hunting): picked mon.noname-c con 0x7f5efc09ac40 addr v1:172.21.15.196:6789/0 2024-03-29T10:40:40.994 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.001+0000 7f5f01a43640 10 monclient(hunting): picked mon.noname-b con 0x7f5efc09c870 addr v1:172.21.15.88:6790/0 2024-03-29T10:40:40.994 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.001+0000 7f5f01a43640 10 monclient(hunting): picked mon.noname-a con 0x7f5efc0b4360 addr v1:172.21.15.88:6789/0 2024-03-29T10:40:40.994 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.001+0000 7f5f01a43640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5efc09dec0 con 0x7f5efc0b4360 2024-03-29T10:40:40.994 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.001+0000 7f5f01a43640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5efc09da30 con 0x7f5efc09ac40 2024-03-29T10:40:40.994 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.001+0000 7f5f01a43640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5efc0b39c0 con 0x7f5efc09c870 2024-03-29T10:40:40.994 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.001+0000 7f5f01a43640 10 monclient(hunting): _renew_subs 2024-03-29T10:40:40.994 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.001+0000 7f5f01a43640 10 monclient(hunting): authenticate will time out at 787.316244s 2024-03-29T10:40:40.994 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.001+0000 7f5efaffd640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f5efc09ac40 0x7f5efc0b4a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.196:39872/0 (socket says 172.21.15.196:39872) 2024-03-29T10:40:40.994 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.001+0000 7f5efaffd640 1 -- 172.21.15.196:0/2241820931 learned_addr learned my addr 172.21.15.196:0/2241820931 (peer_addr_for_me v1:172.21.15.196:0/0) 2024-03-29T10:40:40.994 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.001+0000 7f5ef9ffb640 1 -- 172.21.15.196:0/2241820931 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3790316738 0 0) 0x7f5efc09da30 con 0x7f5efc09ac40 2024-03-29T10:40:40.994 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.001+0000 7f5ef9ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:40:40.995 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.001+0000 7f5ef9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:40.995 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.001+0000 7f5ef9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:40.995 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.001+0000 7f5ef9ffb640 10 monclient(hunting): my global_id is 4108 2024-03-29T10:40:40.995 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.001+0000 7f5ef9ffb640 1 -- 172.21.15.196:0/2241820931 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ee4003520 con 0x7f5efc09ac40 2024-03-29T10:40:40.995 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.001+0000 7f5ef9ffb640 1 -- 172.21.15.196:0/2241820931 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1154431101 0 0) 0x7f5efc0b39c0 con 0x7f5efc09c870 2024-03-29T10:40:40.995 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.001+0000 7f5ef9ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:40:40.995 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.001+0000 7f5ef9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:40.995 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.001+0000 7f5ef9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:40.995 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.001+0000 7f5ef9ffb640 10 monclient(hunting): my global_id is 4106 2024-03-29T10:40:40.995 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.001+0000 7f5ef9ffb640 1 -- 172.21.15.196:0/2241820931 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5efc09da30 con 0x7f5efc09c870 2024-03-29T10:40:40.995 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.001+0000 7f5ef9ffb640 1 -- 172.21.15.196:0/2241820931 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1589388464 0 0) 0x7f5ef40013c0 con 0x7f5efc0b4360 2024-03-29T10:40:40.995 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.001+0000 7f5ef9ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:40:40.995 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.001+0000 7f5ef9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:40.995 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.001+0000 7f5ef9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:40.995 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.001+0000 7f5ef9ffb640 10 monclient(hunting): my global_id is 4110 2024-03-29T10:40:40.996 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.001+0000 7f5ef9ffb640 1 -- 172.21.15.196:0/2241820931 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5efc0b39c0 con 0x7f5efc0b4360 2024-03-29T10:40:40.996 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.001+0000 7f5ef9ffb640 1 -- 172.21.15.196:0/2241820931 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1391920902 0 0) 0x7f5ee4003520 con 0x7f5efc09ac40 2024-03-29T10:40:40.996 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.001+0000 7f5ef9ffb640 1 -- 172.21.15.196:0/2241820931 >> v1:172.21.15.88:6790/0 conn(0x7f5efc09c870 legacy=0x7f5efc13f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:40.996 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.001+0000 7f5ef9ffb640 1 -- 172.21.15.196:0/2241820931 >> v1:172.21.15.88:6789/0 conn(0x7f5efc0b4360 legacy=0x7f5efc14d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:40.996 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.001+0000 7f5ef9ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:40:40.996 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.001+0000 7f5ef9ffb640 1 monclient: found mon.noname-c 2024-03-29T10:40:40.996 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.001+0000 7f5ef9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:40:40.996 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.001+0000 7f5ef9ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.196:6789/0 2024-03-29T10:40:40.996 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.001+0000 7f5ef9ffb640 1 -- 172.21.15.196:0/2241820931 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5efc0b5160 con 0x7f5efc09ac40 2024-03-29T10:40:40.996 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.001+0000 7f5ef9ffb640 10 monclient: _finish_auth 0 2024-03-29T10:40:40.996 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.001+0000 7f5ef9ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:40:40.996 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.001+0000 7f5ef9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:40:40.996 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.001+0000 7f5ef9ffb640 1 -- 172.21.15.196:0/2241820931 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f5ef0001610 con 0x7f5efc09ac40 2024-03-29T10:40:40.996 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.001+0000 7f5f01a43640 5 monclient: authenticate success, global_id 4108 2024-03-29T10:40:40.997 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.001+0000 7f5f01a43640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:40:40.997 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.001+0000 7f5ef9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:40.997 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.001+0000 7f5ef9ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:40:40.997 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.001+0000 7f5ef9ffb640 10 monclient: dump: 2024-03-29T10:40:40.997 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 1 2024-03-29T10:40:40.997 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:40:40.997 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:40:40.997 INFO:teuthology.orchestra.run.smithi196.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:40:40.997 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:40.997 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 1 2024-03-29T10:40:40.997 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:40:40.997 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:40:40.997 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:40:40.997 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-29T10:40:40.998 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.001+0000 7f5f01a43640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:40:40.998 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.001+0000 7f5ef9ffb640 1 -- 172.21.15.196:0/2241820931 <== mon.1 v1:172.21.15.196:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5ef0001770 con 0x7f5efc09ac40 2024-03-29T10:40:40.998 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.001+0000 7f5ef9ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:40:40.998 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.001+0000 7f5ef9ffb640 1 -- 172.21.15.196:0/2241820931 <== mon.1 v1:172.21.15.196:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f5ef0001bc0 con 0x7f5efc09ac40 2024-03-29T10:40:40.998 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.001+0000 7f5f01a43640 10 monclient: get_monmap_and_config success 2024-03-29T10:40:40.998 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.001+0000 7f5f01a43640 4 set_mon_vals no callback set 2024-03-29T10:40:40.999 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.001+0000 7f5ef9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:40.999 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.001+0000 7f5ef9ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:40:40.999 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.001+0000 7f5ef9ffb640 10 monclient: dump: 2024-03-29T10:40:40.999 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 1 2024-03-29T10:40:40.999 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:40:40.999 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:40:40.999 INFO:teuthology.orchestra.run.smithi196.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:40:40.999 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:40.999 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 1 2024-03-29T10:40:40.999 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:40:40.999 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:40:40.999 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:40:40.999 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-29T10:40:40.999 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.002+0000 7f5f01a43640 10 monclient: shutdown 2024-03-29T10:40:40.999 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.002+0000 7f5f01a43640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:40:41.000 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.002+0000 7f5f01a43640 1 -- 172.21.15.196:0/2241820931 >> v1:172.21.15.196:6789/0 conn(0x7f5efc09ac40 legacy=0x7f5efc0b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:41.000 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.002+0000 7f5f01a43640 1 -- 172.21.15.196:0/2241820931 shutdown_connections 2024-03-29T10:40:41.000 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.002+0000 7f5f01a43640 1 -- 172.21.15.196:0/2241820931 >> 172.21.15.196:0/2241820931 conn(0x7f5efc066b00 msgr2=0x7f5efc066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:40:41.000 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.002+0000 7f5f01a43640 1 -- 172.21.15.196:0/2241820931 shutdown_connections 2024-03-29T10:40:41.000 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.002+0000 7f5f01a43640 1 -- 172.21.15.196:0/2241820931 wait complete. 2024-03-29T10:40:41.000 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.003+0000 7f5f01a43640 10 monclient: build_initial_monmap 2024-03-29T10:40:41.000 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.003+0000 7f5f01a43640 10 monclient: monmap: 2024-03-29T10:40:41.000 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 0 2024-03-29T10:40:41.000 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:40:41.000 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 0.000000 2024-03-29T10:40:41.000 INFO:teuthology.orchestra.run.smithi196.stderr:created 0.000000 2024-03-29T10:40:41.000 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 0 (unknown) 2024-03-29T10:40:41.000 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 1 2024-03-29T10:40:41.001 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-03-29T10:40:41.001 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-29T10:40:41.001 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.88:6790/0 mon.noname-c 2024-03-29T10:40:41.001 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-29T10:40:41.001 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.003+0000 7f5f01a43640 1 librados: starting msgr at 2024-03-29T10:40:41.001 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.003+0000 7f5f01a43640 1 librados: starting objecter 2024-03-29T10:40:41.001 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.003+0000 7f5f01a43640 1 Processor -- start 2024-03-29T10:40:41.001 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.003+0000 7f5f01a43640 1 -- start start 2024-03-29T10:40:41.001 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.003+0000 7f5f01a43640 1 librados: setting wanted keys 2024-03-29T10:40:41.001 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.003+0000 7f5f01a43640 1 librados: calling monclient init 2024-03-29T10:40:41.001 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.003+0000 7f5f01a43640 10 monclient: init 2024-03-29T10:40:41.001 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.004+0000 7f5f01a43640 10 monclient: _reopen_session rank -1 2024-03-29T10:40:41.001 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.004+0000 7f5f01a43640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:40:41.001 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.004+0000 7f5f01a43640 10 monclient(hunting): picked mon.noname-c con 0x7f5efc09ac40 addr v1:172.21.15.88:6790/0 2024-03-29T10:40:41.001 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.004+0000 7f5f01a43640 10 monclient(hunting): picked mon.noname-b con 0x7f5efc1d7f90 addr v1:172.21.15.196:6789/0 2024-03-29T10:40:41.002 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.004+0000 7f5f01a43640 10 monclient(hunting): picked mon.noname-a con 0x7f5efc0b08b0 addr v1:172.21.15.88:6789/0 2024-03-29T10:40:41.002 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.004+0000 7f5f01a43640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5efc0b18d0 con 0x7f5efc0b08b0 2024-03-29T10:40:41.002 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.004+0000 7f5f01a43640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5efc1d7380 con 0x7f5efc1d7f90 2024-03-29T10:40:41.002 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.004+0000 7f5f01a43640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5efc0af7c0 con 0x7f5efc09ac40 2024-03-29T10:40:41.002 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.004+0000 7f5f01a43640 10 monclient(hunting): _renew_subs 2024-03-29T10:40:41.002 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.004+0000 7f5f01a43640 10 monclient(hunting): authenticate will time out at 787.319201s 2024-03-29T10:40:41.002 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.004+0000 7f5efa7fc640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f5efc1d7f90 0x7f5efc0b01a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.196:39876/0 (socket says 172.21.15.196:39876) 2024-03-29T10:40:41.002 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.004+0000 7f5efa7fc640 1 -- 172.21.15.196:0/3350430564 learned_addr learned my addr 172.21.15.196:0/3350430564 (peer_addr_for_me v1:172.21.15.196:0/0) 2024-03-29T10:40:41.002 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.004+0000 7f5ee3fff640 10 client.?.objecter ms_handle_connect 0x7f5efc1d7f90 2024-03-29T10:40:41.002 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.004+0000 7f5ee3fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:40:41.002 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.004+0000 7f5ee3fff640 10 client.?.objecter ms_handle_connect 0x7f5efc09ac40 2024-03-29T10:40:41.002 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.004+0000 7f5ee3fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:40:41.002 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.004+0000 7f5ee3fff640 10 client.?.objecter ms_handle_connect 0x7f5efc0b08b0 2024-03-29T10:40:41.002 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.004+0000 7f5ee3fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:40:41.003 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.004+0000 7f5ee3fff640 1 -- 172.21.15.196:0/3350430564 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1825732277 0 0) 0x7f5efc1d7380 con 0x7f5efc1d7f90 2024-03-29T10:40:41.003 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.004+0000 7f5ee3fff640 10 monclient(hunting): _negotiate 2024-03-29T10:40:41.003 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.004+0000 7f5ee3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:41.003 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.004+0000 7f5ee3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:41.003 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.004+0000 7f5ee3fff640 10 monclient(hunting): my global_id is 4111 2024-03-29T10:40:41.003 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.004+0000 7f5ee3fff640 1 -- 172.21.15.196:0/3350430564 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ee8004eb0 con 0x7f5efc1d7f90 2024-03-29T10:40:41.003 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.004+0000 7f5ee3fff640 1 -- 172.21.15.196:0/3350430564 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2125166044 0 0) 0x7f5efc0af7c0 con 0x7f5efc09ac40 2024-03-29T10:40:41.003 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.004+0000 7f5ee3fff640 10 monclient(hunting): _negotiate 2024-03-29T10:40:41.003 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.004+0000 7f5ee3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:41.003 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.004+0000 7f5ee3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:41.003 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.004+0000 7f5ee3fff640 10 monclient(hunting): my global_id is 4109 2024-03-29T10:40:41.003 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.004+0000 7f5ee3fff640 1 -- 172.21.15.196:0/3350430564 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5efc1d7380 con 0x7f5efc09ac40 2024-03-29T10:40:41.003 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.004+0000 7f5ee3fff640 1 -- 172.21.15.196:0/3350430564 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1699317107 0 0) 0x7f5efc0b18d0 con 0x7f5efc0b08b0 2024-03-29T10:40:41.003 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.004+0000 7f5ee3fff640 10 monclient(hunting): _negotiate 2024-03-29T10:40:41.004 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.004+0000 7f5ee3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:41.004 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.004+0000 7f5ee3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:41.004 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.004+0000 7f5ee3fff640 10 monclient(hunting): my global_id is 4113 2024-03-29T10:40:41.004 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.004+0000 7f5ee3fff640 1 -- 172.21.15.196:0/3350430564 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5efc0af7c0 con 0x7f5efc0b08b0 2024-03-29T10:40:41.004 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.004+0000 7f5ee3fff640 1 -- 172.21.15.196:0/3350430564 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1676031491 0 0) 0x7f5ee8004eb0 con 0x7f5efc1d7f90 2024-03-29T10:40:41.004 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.004+0000 7f5ee3fff640 1 -- 172.21.15.196:0/3350430564 --> v1:172.21.15.196:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5efc0b18d0 con 0x7f5efc1d7f90 2024-03-29T10:40:41.004 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.004+0000 7f5ee3fff640 1 -- 172.21.15.196:0/3350430564 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f5eec001a50 con 0x7f5efc1d7f90 2024-03-29T10:40:41.004 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.004+0000 7f5ee3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:41.004 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.004+0000 7f5ee3fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:40:41.004 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.004+0000 7f5ee3fff640 10 monclient(hunting): dump: 2024-03-29T10:40:41.004 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 1 2024-03-29T10:40:41.005 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:40:41.005 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:40:41.005 INFO:teuthology.orchestra.run.smithi196.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:40:41.005 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:41.005 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 1 2024-03-29T10:40:41.005 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:40:41.005 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:40:41.005 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:40:41.005 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-29T10:40:41.005 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.004+0000 7f5ee3fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:40:41.005 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.004+0000 7f5ee3fff640 1 -- 172.21.15.196:0/3350430564 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2566557320 0 0) 0x7f5efc1d7380 con 0x7f5efc09ac40 2024-03-29T10:40:41.005 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.004+0000 7f5ee3fff640 1 -- 172.21.15.196:0/3350430564 --> v1:172.21.15.88:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5ee8004eb0 con 0x7f5efc09ac40 2024-03-29T10:40:41.005 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.004+0000 7f5ee3fff640 1 -- 172.21.15.196:0/3350430564 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f5ef0001a80 con 0x7f5efc09ac40 2024-03-29T10:40:41.005 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.004+0000 7f5ee3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:41.006 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.004+0000 7f5ee3fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:40:41.006 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.004+0000 7f5ee3fff640 10 monclient(hunting): dump: 2024-03-29T10:40:41.006 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 1 2024-03-29T10:40:41.006 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:40:41.006 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:40:41.006 INFO:teuthology.orchestra.run.smithi196.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:40:41.006 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:41.006 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 1 2024-03-29T10:40:41.006 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:40:41.006 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:40:41.006 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:40:41.006 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-29T10:40:41.006 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.004+0000 7f5ee3fff640 1 -- 172.21.15.196:0/3350430564 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 390997161 0 0) 0x7f5efc0af7c0 con 0x7f5efc0b08b0 2024-03-29T10:40:41.006 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.004+0000 7f5ee3fff640 1 -- 172.21.15.196:0/3350430564 --> v1:172.21.15.88:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5efc1d7380 con 0x7f5efc0b08b0 2024-03-29T10:40:41.007 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.004+0000 7f5ee3fff640 1 -- 172.21.15.196:0/3350430564 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f5ef4001820 con 0x7f5efc0b08b0 2024-03-29T10:40:41.007 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.004+0000 7f5ee3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:41.007 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.004+0000 7f5ee3fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:40:41.007 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.004+0000 7f5ee3fff640 10 monclient(hunting): dump: 2024-03-29T10:40:41.007 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 1 2024-03-29T10:40:41.007 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:40:41.007 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:40:41.007 INFO:teuthology.orchestra.run.smithi196.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:40:41.007 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:41.007 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 1 2024-03-29T10:40:41.007 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:40:41.007 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:40:41.007 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:40:41.007 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-29T10:40:41.008 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.004+0000 7f5ee3fff640 1 -- 172.21.15.196:0/3350430564 <== mon.1 v1:172.21.15.196:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 628966846 0 0) 0x7f5efc0b18d0 con 0x7f5efc1d7f90 2024-03-29T10:40:41.008 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.004+0000 7f5ee3fff640 1 -- 172.21.15.196:0/3350430564 >> v1:172.21.15.88:6790/0 conn(0x7f5efc09ac40 legacy=0x7f5efc1d7880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:41.008 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.004+0000 7f5ee3fff640 1 -- 172.21.15.196:0/3350430564 >> v1:172.21.15.88:6789/0 conn(0x7f5efc0b08b0 legacy=0x7f5efc139010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:41.008 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.004+0000 7f5ee3fff640 10 monclient: _finish_hunting 0 2024-03-29T10:40:41.008 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.004+0000 7f5ee3fff640 1 monclient: found mon.b 2024-03-29T10:40:41.008 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.004+0000 7f5ee3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:40:41.008 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.004+0000 7f5ee3fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T10:40:41.008 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.004+0000 7f5ee3fff640 1 -- 172.21.15.196:0/3350430564 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5efc0acf60 con 0x7f5efc1d7f90 2024-03-29T10:40:41.008 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.004+0000 7f5ee3fff640 10 monclient: _finish_auth 0 2024-03-29T10:40:41.008 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.004+0000 7f5ee3fff640 10 monclient: _check_auth_tickets 2024-03-29T10:40:41.008 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.004+0000 7f5ee3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:40:41.008 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.004+0000 7f5f01a43640 5 monclient: authenticate success, global_id 4111 2024-03-29T10:40:41.008 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.004+0000 7f5f01a43640 10 monclient: _renew_subs 2024-03-29T10:40:41.008 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.004+0000 7f5f01a43640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T10:40:41.009 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.004+0000 7f5f01a43640 1 -- 172.21.15.196:0/3350430564 --> v1:172.21.15.196:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5efc139720 con 0x7f5efc1d7f90 2024-03-29T10:40:41.009 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.005+0000 7f5f01a43640 10 client.4111.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:40:41.009 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.005+0000 7f5f01a43640 10 monclient: _renew_subs 2024-03-29T10:40:41.009 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.005+0000 7f5f01a43640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T10:40:41.009 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.005+0000 7f5f01a43640 1 -- 172.21.15.196:0/3350430564 --> v1:172.21.15.196:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5efc139ca0 con 0x7f5efc1d7f90 2024-03-29T10:40:41.009 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.005+0000 7f5f01a43640 1 librados: init done 2024-03-29T10:40:41.009 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.005+0000 7f5ee3fff640 1 -- 172.21.15.196:0/3350430564 <== mon.1 v1:172.21.15.196:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5eec001c90 con 0x7f5efc1d7f90 2024-03-29T10:40:41.009 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.005+0000 7f5ee3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:40:41.009 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.005+0000 7f5ee3fff640 1 -- 172.21.15.196:0/3350430564 <== mon.1 v1:172.21.15.196:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f5eec003a60 con 0x7f5efc1d7f90 2024-03-29T10:40:41.009 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.005+0000 7f5ee3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:41.009 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.005+0000 7f5ef97fa640 4 set_mon_vals no callback set 2024-03-29T10:40:41.009 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.005+0000 7f5ee3fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:40:41.009 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.005+0000 7f5ee3fff640 10 monclient: dump: 2024-03-29T10:40:41.009 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 1 2024-03-29T10:40:41.010 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:40:41.010 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:40:41.010 INFO:teuthology.orchestra.run.smithi196.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:40:41.010 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:41.010 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 1 2024-03-29T10:40:41.010 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:40:41.010 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:40:41.010 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:40:41.010 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-29T10:40:41.010 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.005+0000 7f5ee3fff640 1 -- 172.21.15.196:0/3350430564 <== mon.1 v1:172.21.15.196:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f5eec004010 con 0x7f5efc1d7f90 2024-03-29T10:40:41.010 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.005+0000 7f5ee3fff640 1 -- 172.21.15.196:0/3350430564 <== mon.1 v1:172.21.15.196:6789/0 8 ==== osd_map(2..2 src has 1..2) v4 ==== 725+0+0 (unknown 1452808794 0 0) 0x7f5eec0066d0 con 0x7f5efc1d7f90 2024-03-29T10:40:41.010 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.005+0000 7f5ee3fff640 10 client.4111.objecter ms_dispatch 0x7f5efc000b90 osd_map(2..2 src has 1..2) v4 2024-03-29T10:40:41.010 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.005+0000 7f5ee3fff640 3 client.4111.objecter handle_osd_map got epochs [2,2] > 0 2024-03-29T10:40:41.010 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.005+0000 7f5ee3fff640 3 client.4111.objecter handle_osd_map decoding full epoch 2 2024-03-29T10:40:41.011 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.005+0000 7f5ee3fff640 20 client.4111.objecter dump_active .. 0 homeless 2024-03-29T10:40:41.011 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.005+0000 7f5f01a43640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:40:41.011 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.005+0000 7f5f01a43640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:40:41.011 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.005+0000 7f5f01a43640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T10:40:41.011 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.005+0000 7f5f01a43640 1 -- 172.21.15.196:0/3350430564 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5edc005080 con 0x7f5efc1d7f90 2024-03-29T10:40:41.011 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.008+0000 7f5ee3fff640 1 -- 172.21.15.196:0/3350430564 <== mon.1 v1:172.21.15.196:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f5eec004450 con 0x7f5efc1d7f90 2024-03-29T10:40:41.011 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.008+0000 7f5ee3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:40:41.011 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.008+0000 7f5ee3fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:40:41.063 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.079+0000 7f5f01a43640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "a26c0c1c-8de0-438c-a3f6-3df10ea61574", "id": 0}] 2024-03-29T10:40:41.063 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.079+0000 7f5f01a43640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "a26c0c1c-8de0-438c-a3f6-3df10ea61574", "id": 0}] 2024-03-29T10:40:41.063 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.079+0000 7f5f01a43640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T10:40:41.063 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.079+0000 7f5f01a43640 1 -- 172.21.15.196:0/3350430564 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "osd new", "uuid": "a26c0c1c-8de0-438c-a3f6-3df10ea61574", "id": 0} v 0) v1 -- 0x7f5edc0058b0 con 0x7f5efc1d7f90 2024-03-29T10:40:41.069 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.085+0000 7f5ee3fff640 1 -- 172.21.15.196:0/3350430564 <== mon.1 v1:172.21.15.196:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "a26c0c1c-8de0-438c-a3f6-3df10ea61574", "id": 0}]=0 v3) v1 ==== 112+0+2 (unknown 179298789 0 732707515) 0x7f5eec006a30 con 0x7f5efc1d7f90 2024-03-29T10:40:41.069 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.085+0000 7f5ee3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "a26c0c1c-8de0-438c-a3f6-3df10ea61574", "id": 0}] 2024-03-29T10:40:41.069 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.085+0000 7f5ee3fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:40:41.069 INFO:teuthology.orchestra.run.smithi196.stdout:0 2024-03-29T10:40:41.070 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.086+0000 7f5f01a43640 10 librados: watch_flush enter 2024-03-29T10:40:41.070 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.086+0000 7f5f01a43640 10 librados: watch_flush exit 2024-03-29T10:40:41.071 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.086+0000 7f5f01a43640 20 client.4111.objecter shutdown clearing up homeless session... 2024-03-29T10:40:41.071 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.086+0000 7f5f01a43640 10 client.4111.objecter successfully canceled tick 2024-03-29T10:40:41.071 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.086+0000 7f5f01a43640 10 monclient: shutdown 2024-03-29T10:40:41.071 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.086+0000 7f5f01a43640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:40:41.071 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.086+0000 7f5f01a43640 1 -- 172.21.15.196:0/3350430564 >> v1:172.21.15.196:6789/0 conn(0x7f5efc1d7f90 legacy=0x7f5efc0b01a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:41.071 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.086+0000 7f5f01a43640 1 -- 172.21.15.196:0/3350430564 shutdown_connections 2024-03-29T10:40:41.071 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.086+0000 7f5f01a43640 1 -- 172.21.15.196:0/3350430564 >> 172.21.15.196:0/3350430564 conn(0x7f5efc066b00 msgr2=0x7f5efc1420b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:40:41.071 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.086+0000 7f5f01a43640 1 -- 172.21.15.196:0/3350430564 shutdown_connections 2024-03-29T10:40:41.071 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.086+0000 7f5f01a43640 1 -- 172.21.15.196:0/3350430564 wait complete. 2024-03-29T10:40:41.071 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.086+0000 7f5f01a43640 1 librados: shutdown 2024-03-29T10:40:41.080 DEBUG:teuthology.orchestra.run.smithi196:> sudo ceph --cluster ceph osd new 5de24915-8dec-49b6-8b16-f3170197c591 1 2024-03-29T10:40:41.180 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.196+0000 7f76887ba640 10 monclient: get_monmap_and_config 2024-03-29T10:40:41.180 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.196+0000 7f76887ba640 10 monclient: build_initial_monmap 2024-03-29T10:40:41.180 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.196+0000 7f76887ba640 10 monclient: monmap: 2024-03-29T10:40:41.180 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 0 2024-03-29T10:40:41.180 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:40:41.180 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 2024-03-29T10:40:41.197579+0000 2024-03-29T10:40:41.180 INFO:teuthology.orchestra.run.smithi196.stderr:created 2024-03-29T10:40:41.197579+0000 2024-03-29T10:40:41.180 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 0 (unknown) 2024-03-29T10:40:41.180 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 1 2024-03-29T10:40:41.180 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-03-29T10:40:41.181 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-29T10:40:41.181 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.88:6790/0 mon.noname-b 2024-03-29T10:40:41.181 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-29T10:40:41.191 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.197+0000 7f76887ba640 1 Processor -- start 2024-03-29T10:40:41.191 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.197+0000 7f76887ba640 1 -- start start 2024-03-29T10:40:41.191 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.197+0000 7f76887ba640 10 monclient: init 2024-03-29T10:40:41.191 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.197+0000 7f76887ba640 10 monclient: _reopen_session rank -1 2024-03-29T10:40:41.191 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.197+0000 7f76887ba640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:40:41.191 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.198+0000 7f76887ba640 10 monclient(hunting): picked mon.noname-a con 0x7f768009ac40 addr v1:172.21.15.88:6789/0 2024-03-29T10:40:41.191 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.198+0000 7f76887ba640 10 monclient(hunting): picked mon.noname-c con 0x7f768009c870 addr v1:172.21.15.196:6789/0 2024-03-29T10:40:41.191 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.198+0000 7f76887ba640 10 monclient(hunting): picked mon.noname-b con 0x7f76800b4360 addr v1:172.21.15.88:6790/0 2024-03-29T10:40:41.191 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.198+0000 7f76887ba640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f768009dec0 con 0x7f768009ac40 2024-03-29T10:40:41.191 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.198+0000 7f76887ba640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f768009da30 con 0x7f768009c870 2024-03-29T10:40:41.191 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.198+0000 7f76887ba640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76800b39c0 con 0x7f76800b4360 2024-03-29T10:40:41.192 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.198+0000 7f76887ba640 10 monclient(hunting): _renew_subs 2024-03-29T10:40:41.192 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.198+0000 7f76887ba640 10 monclient(hunting): authenticate will time out at 787.513065s 2024-03-29T10:40:41.192 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.198+0000 7f7685d55640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f768009c870 0x7f768013f3a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.196:39890/0 (socket says 172.21.15.196:39890) 2024-03-29T10:40:41.192 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.198+0000 7f7685d55640 1 -- 172.21.15.196:0/3298318918 learned_addr learned my addr 172.21.15.196:0/3298318918 (peer_addr_for_me v1:172.21.15.196:0/0) 2024-03-29T10:40:41.192 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.198+0000 7f7685554640 1 -- 172.21.15.196:0/3298318918 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2353639050 0 0) 0x7f768009da30 con 0x7f768009c870 2024-03-29T10:40:41.192 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.198+0000 7f7685554640 10 monclient(hunting): _negotiate 2024-03-29T10:40:41.192 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.198+0000 7f7685554640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:41.192 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.198+0000 7f7685554640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:41.192 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.198+0000 7f7685554640 10 monclient(hunting): my global_id is 4114 2024-03-29T10:40:41.192 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.198+0000 7f7685554640 1 -- 172.21.15.196:0/3298318918 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f766c003520 con 0x7f768009c870 2024-03-29T10:40:41.192 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.198+0000 7f7685554640 1 -- 172.21.15.196:0/3298318918 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4185557294 0 0) 0x7f76800b39c0 con 0x7f76800b4360 2024-03-29T10:40:41.192 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.198+0000 7f7685554640 10 monclient(hunting): _negotiate 2024-03-29T10:40:41.192 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.198+0000 7f7685554640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:41.192 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.198+0000 7f7685554640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:41.193 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.198+0000 7f7685554640 10 monclient(hunting): my global_id is 4112 2024-03-29T10:40:41.193 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.198+0000 7f7685554640 1 -- 172.21.15.196:0/3298318918 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f768009da30 con 0x7f76800b4360 2024-03-29T10:40:41.193 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.198+0000 7f7685554640 1 -- 172.21.15.196:0/3298318918 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4019749718 0 0) 0x7f767c001110 con 0x7f768009ac40 2024-03-29T10:40:41.193 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.198+0000 7f7685554640 10 monclient(hunting): _negotiate 2024-03-29T10:40:41.193 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.198+0000 7f7685554640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:41.193 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.198+0000 7f7685554640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:41.193 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.198+0000 7f7685554640 10 monclient(hunting): my global_id is 4116 2024-03-29T10:40:41.193 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.198+0000 7f7685554640 1 -- 172.21.15.196:0/3298318918 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f76800b39c0 con 0x7f768009ac40 2024-03-29T10:40:41.193 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.198+0000 7f7685554640 1 -- 172.21.15.196:0/3298318918 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 95481243 0 0) 0x7f766c003520 con 0x7f768009c870 2024-03-29T10:40:41.193 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.198+0000 7f7685554640 1 -- 172.21.15.196:0/3298318918 >> v1:172.21.15.88:6790/0 conn(0x7f76800b4360 legacy=0x7f768014d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:41.193 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.198+0000 7f7685554640 1 -- 172.21.15.196:0/3298318918 >> v1:172.21.15.88:6789/0 conn(0x7f768009ac40 legacy=0x7f76800b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:41.193 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.198+0000 7f7685554640 10 monclient: _finish_hunting 0 2024-03-29T10:40:41.193 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.198+0000 7f7685554640 1 monclient: found mon.noname-c 2024-03-29T10:40:41.193 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.198+0000 7f7685554640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:40:41.193 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.198+0000 7f7685554640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.196:6789/0 2024-03-29T10:40:41.194 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.198+0000 7f7685554640 1 -- 172.21.15.196:0/3298318918 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76800b5160 con 0x7f768009c870 2024-03-29T10:40:41.194 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.198+0000 7f7685554640 10 monclient: _finish_auth 0 2024-03-29T10:40:41.194 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.198+0000 7f7685554640 10 monclient: _check_auth_tickets 2024-03-29T10:40:41.194 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.198+0000 7f7685554640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:40:41.194 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.198+0000 7f7685554640 1 -- 172.21.15.196:0/3298318918 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f7670001690 con 0x7f768009c870 2024-03-29T10:40:41.194 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.198+0000 7f76887ba640 5 monclient: authenticate success, global_id 4114 2024-03-29T10:40:41.194 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.198+0000 7f76887ba640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:40:41.194 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.198+0000 7f7685554640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:41.194 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.198+0000 7f7685554640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:40:41.194 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.198+0000 7f7685554640 10 monclient: dump: 2024-03-29T10:40:41.194 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 1 2024-03-29T10:40:41.194 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:40:41.194 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:40:41.194 INFO:teuthology.orchestra.run.smithi196.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:40:41.195 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:41.195 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 1 2024-03-29T10:40:41.195 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:40:41.195 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:40:41.195 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:40:41.195 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-29T10:40:41.195 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.198+0000 7f76887ba640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:40:41.195 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.198+0000 7f7685554640 1 -- 172.21.15.196:0/3298318918 <== mon.1 v1:172.21.15.196:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7670001870 con 0x7f768009c870 2024-03-29T10:40:41.195 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.198+0000 7f7685554640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:40:41.195 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.198+0000 7f7685554640 1 -- 172.21.15.196:0/3298318918 <== mon.1 v1:172.21.15.196:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f7670001c80 con 0x7f768009c870 2024-03-29T10:40:41.195 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.198+0000 7f76887ba640 10 monclient: get_monmap_and_config success 2024-03-29T10:40:41.195 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.198+0000 7f76887ba640 4 set_mon_vals no callback set 2024-03-29T10:40:41.195 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.198+0000 7f7685554640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:41.195 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.198+0000 7f7685554640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:40:41.195 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.198+0000 7f7685554640 10 monclient: dump: 2024-03-29T10:40:41.196 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 1 2024-03-29T10:40:41.196 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:40:41.196 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:40:41.196 INFO:teuthology.orchestra.run.smithi196.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:40:41.196 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:41.196 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 1 2024-03-29T10:40:41.196 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:40:41.196 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:40:41.196 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:40:41.196 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-29T10:40:41.196 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.199+0000 7f76887ba640 10 monclient: shutdown 2024-03-29T10:40:41.196 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.199+0000 7f76887ba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:40:41.196 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.199+0000 7f76887ba640 1 -- 172.21.15.196:0/3298318918 >> v1:172.21.15.196:6789/0 conn(0x7f768009c870 legacy=0x7f768013f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:41.196 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.199+0000 7f76887ba640 1 -- 172.21.15.196:0/3298318918 shutdown_connections 2024-03-29T10:40:41.197 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.199+0000 7f76887ba640 1 -- 172.21.15.196:0/3298318918 >> 172.21.15.196:0/3298318918 conn(0x7f7680066b00 msgr2=0x7f7680066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:40:41.197 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.199+0000 7f76887ba640 1 -- 172.21.15.196:0/3298318918 shutdown_connections 2024-03-29T10:40:41.197 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.199+0000 7f76887ba640 1 -- 172.21.15.196:0/3298318918 wait complete. 2024-03-29T10:40:41.197 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.199+0000 7f76887ba640 10 monclient: build_initial_monmap 2024-03-29T10:40:41.197 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.199+0000 7f76887ba640 10 monclient: monmap: 2024-03-29T10:40:41.197 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 0 2024-03-29T10:40:41.197 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:40:41.197 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 0.000000 2024-03-29T10:40:41.197 INFO:teuthology.orchestra.run.smithi196.stderr:created 0.000000 2024-03-29T10:40:41.197 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 0 (unknown) 2024-03-29T10:40:41.197 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 1 2024-03-29T10:40:41.197 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-03-29T10:40:41.197 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-29T10:40:41.197 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.88:6790/0 mon.noname-c 2024-03-29T10:40:41.198 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-29T10:40:41.198 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.200+0000 7f76887ba640 1 librados: starting msgr at 2024-03-29T10:40:41.198 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.200+0000 7f76887ba640 1 librados: starting objecter 2024-03-29T10:40:41.198 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.200+0000 7f76887ba640 1 Processor -- start 2024-03-29T10:40:41.198 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.200+0000 7f76887ba640 1 -- start start 2024-03-29T10:40:41.198 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.200+0000 7f76887ba640 1 librados: setting wanted keys 2024-03-29T10:40:41.198 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.200+0000 7f76887ba640 1 librados: calling monclient init 2024-03-29T10:40:41.198 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.200+0000 7f76887ba640 10 monclient: init 2024-03-29T10:40:41.198 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.200+0000 7f76887ba640 10 monclient: _reopen_session rank -1 2024-03-29T10:40:41.198 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.200+0000 7f76887ba640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:40:41.198 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.200+0000 7f76887ba640 10 monclient(hunting): picked mon.noname-c con 0x7f768009ac40 addr v1:172.21.15.88:6790/0 2024-03-29T10:40:41.198 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.200+0000 7f76887ba640 10 monclient(hunting): picked mon.noname-a con 0x7f76801db230 addr v1:172.21.15.88:6789/0 2024-03-29T10:40:41.198 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.200+0000 7f76887ba640 10 monclient(hunting): picked mon.noname-b con 0x7f76800b0910 addr v1:172.21.15.196:6789/0 2024-03-29T10:40:41.198 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.200+0000 7f76887ba640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76801da690 con 0x7f76801db230 2024-03-29T10:40:41.198 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.200+0000 7f76887ba640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76800b1910 con 0x7f76800b0910 2024-03-29T10:40:41.199 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.200+0000 7f76887ba640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76800acd00 con 0x7f768009ac40 2024-03-29T10:40:41.199 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.200+0000 7f76887ba640 10 monclient(hunting): _renew_subs 2024-03-29T10:40:41.199 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.200+0000 7f76887ba640 10 monclient(hunting): authenticate will time out at 787.515862s 2024-03-29T10:40:41.199 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.200+0000 7f7686d57640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f76800b0910 0x7f76801e1b70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.196:39902/0 (socket says 172.21.15.196:39902) 2024-03-29T10:40:41.199 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.200+0000 7f7686d57640 1 -- 172.21.15.196:0/962343532 learned_addr learned my addr 172.21.15.196:0/962343532 (peer_addr_for_me v1:172.21.15.196:0/0) 2024-03-29T10:40:41.199 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.201+0000 7f7676ffd640 10 client.?.objecter ms_handle_connect 0x7f76800b0910 2024-03-29T10:40:41.199 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.201+0000 7f7676ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:40:41.199 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.201+0000 7f7676ffd640 10 client.?.objecter ms_handle_connect 0x7f768009ac40 2024-03-29T10:40:41.199 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.201+0000 7f7676ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:40:41.200 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.201+0000 7f7676ffd640 10 client.?.objecter ms_handle_connect 0x7f76801db230 2024-03-29T10:40:41.200 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.201+0000 7f7676ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:40:41.200 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.201+0000 7f7676ffd640 1 -- 172.21.15.196:0/962343532 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 948181459 0 0) 0x7f76800b1910 con 0x7f76800b0910 2024-03-29T10:40:41.200 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.201+0000 7f7676ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:40:41.200 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.201+0000 7f7676ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:41.200 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.201+0000 7f7676ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:41.200 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.201+0000 7f7676ffd640 10 monclient(hunting): my global_id is 4117 2024-03-29T10:40:41.200 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.201+0000 7f7676ffd640 1 -- 172.21.15.196:0/962343532 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7664004eb0 con 0x7f76800b0910 2024-03-29T10:40:41.200 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.201+0000 7f7676ffd640 1 -- 172.21.15.196:0/962343532 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1352040599 0 0) 0x7f76800acd00 con 0x7f768009ac40 2024-03-29T10:40:41.200 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.201+0000 7f7676ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:40:41.201 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.201+0000 7f7676ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:41.201 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.201+0000 7f7676ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:41.201 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.201+0000 7f7676ffd640 10 monclient(hunting): my global_id is 4115 2024-03-29T10:40:41.201 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.201+0000 7f7676ffd640 1 -- 172.21.15.196:0/962343532 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f76800b1910 con 0x7f768009ac40 2024-03-29T10:40:41.201 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.201+0000 7f7676ffd640 1 -- 172.21.15.196:0/962343532 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3613119058 0 0) 0x7f76801da690 con 0x7f76801db230 2024-03-29T10:40:41.201 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.201+0000 7f7676ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:40:41.201 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.201+0000 7f7676ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:41.201 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.201+0000 7f7676ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:41.201 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.201+0000 7f7676ffd640 10 monclient(hunting): my global_id is 4119 2024-03-29T10:40:41.201 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.201+0000 7f7676ffd640 1 -- 172.21.15.196:0/962343532 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f76800acd00 con 0x7f76801db230 2024-03-29T10:40:41.201 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.201+0000 7f7676ffd640 1 -- 172.21.15.196:0/962343532 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4205030501 0 0) 0x7f7664004eb0 con 0x7f76800b0910 2024-03-29T10:40:41.201 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.201+0000 7f7676ffd640 1 -- 172.21.15.196:0/962343532 --> v1:172.21.15.196:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f76801da690 con 0x7f76800b0910 2024-03-29T10:40:41.201 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.201+0000 7f7676ffd640 1 -- 172.21.15.196:0/962343532 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f7678001bb0 con 0x7f76800b0910 2024-03-29T10:40:41.201 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.201+0000 7f7676ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:41.202 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.201+0000 7f7676ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:40:41.202 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.201+0000 7f7676ffd640 10 monclient(hunting): dump: 2024-03-29T10:40:41.202 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 1 2024-03-29T10:40:41.202 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:40:41.202 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:40:41.202 INFO:teuthology.orchestra.run.smithi196.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:40:41.202 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:41.202 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 1 2024-03-29T10:40:41.202 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:40:41.202 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:40:41.202 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:40:41.202 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-29T10:40:41.202 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.201+0000 7f7676ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:40:41.202 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.201+0000 7f7676ffd640 1 -- 172.21.15.196:0/962343532 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 64884916 0 0) 0x7f76800b1910 con 0x7f768009ac40 2024-03-29T10:40:41.203 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.201+0000 7f7676ffd640 1 -- 172.21.15.196:0/962343532 --> v1:172.21.15.88:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7664004eb0 con 0x7f768009ac40 2024-03-29T10:40:41.203 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.201+0000 7f7676ffd640 1 -- 172.21.15.196:0/962343532 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f767c001860 con 0x7f768009ac40 2024-03-29T10:40:41.203 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.201+0000 7f7676ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:41.203 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.201+0000 7f7676ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:40:41.203 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.201+0000 7f7676ffd640 10 monclient(hunting): dump: 2024-03-29T10:40:41.203 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 1 2024-03-29T10:40:41.203 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:40:41.203 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:40:41.203 INFO:teuthology.orchestra.run.smithi196.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:40:41.203 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:41.203 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 1 2024-03-29T10:40:41.203 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:40:41.203 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:40:41.203 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:40:41.203 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-29T10:40:41.204 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.201+0000 7f7676ffd640 1 -- 172.21.15.196:0/962343532 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1894382213 0 0) 0x7f76800acd00 con 0x7f76801db230 2024-03-29T10:40:41.204 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.201+0000 7f7676ffd640 1 -- 172.21.15.196:0/962343532 --> v1:172.21.15.88:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f76800b1910 con 0x7f76801db230 2024-03-29T10:40:41.204 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.201+0000 7f7676ffd640 1 -- 172.21.15.196:0/962343532 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f76700019e0 con 0x7f76801db230 2024-03-29T10:40:41.204 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.201+0000 7f7676ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:41.204 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.201+0000 7f7676ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:40:41.204 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.201+0000 7f7676ffd640 10 monclient(hunting): dump: 2024-03-29T10:40:41.204 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 1 2024-03-29T10:40:41.204 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:40:41.204 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:40:41.204 INFO:teuthology.orchestra.run.smithi196.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:40:41.204 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:41.204 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 1 2024-03-29T10:40:41.204 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:40:41.205 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:40:41.205 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:40:41.205 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-29T10:40:41.205 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.201+0000 7f7676ffd640 1 -- 172.21.15.196:0/962343532 <== mon.1 v1:172.21.15.196:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3536519855 0 0) 0x7f76801da690 con 0x7f76800b0910 2024-03-29T10:40:41.205 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.201+0000 7f7676ffd640 1 -- 172.21.15.196:0/962343532 >> v1:172.21.15.88:6790/0 conn(0x7f768009ac40 legacy=0x7f76801dab20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:41.205 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.201+0000 7f7676ffd640 1 -- 172.21.15.196:0/962343532 >> v1:172.21.15.88:6789/0 conn(0x7f76801db230 legacy=0x7f76800b0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:41.205 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.201+0000 7f7676ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:40:41.205 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.201+0000 7f7676ffd640 1 monclient: found mon.b 2024-03-29T10:40:41.206 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.201+0000 7f7676ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:40:41.206 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.201+0000 7f7676ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T10:40:41.206 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.201+0000 7f7676ffd640 1 -- 172.21.15.196:0/962343532 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76800ace90 con 0x7f76800b0910 2024-03-29T10:40:41.206 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.201+0000 7f7676ffd640 10 monclient: _finish_auth 0 2024-03-29T10:40:41.207 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.201+0000 7f7676ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:40:41.207 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.201+0000 7f7676ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:40:41.207 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.201+0000 7f76887ba640 5 monclient: authenticate success, global_id 4117 2024-03-29T10:40:41.207 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.201+0000 7f76887ba640 10 monclient: _renew_subs 2024-03-29T10:40:41.207 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.201+0000 7f76887ba640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T10:40:41.207 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.201+0000 7f76887ba640 1 -- 172.21.15.196:0/962343532 --> v1:172.21.15.196:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76800af5f0 con 0x7f76800b0910 2024-03-29T10:40:41.207 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.201+0000 7f76887ba640 10 client.4117.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:40:41.207 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.201+0000 7f7676ffd640 1 -- 172.21.15.196:0/962343532 <== mon.1 v1:172.21.15.196:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7678003990 con 0x7f76800b0910 2024-03-29T10:40:41.207 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.201+0000 7f76887ba640 10 monclient: _renew_subs 2024-03-29T10:40:41.207 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.201+0000 7f76887ba640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T10:40:41.207 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.201+0000 7f76887ba640 1 -- 172.21.15.196:0/962343532 --> v1:172.21.15.196:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f76801e3470 con 0x7f76800b0910 2024-03-29T10:40:41.207 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.201+0000 7f76887ba640 1 librados: init done 2024-03-29T10:40:41.208 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.201+0000 7f7676ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:40:41.208 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.201+0000 7f7676ffd640 1 -- 172.21.15.196:0/962343532 <== mon.1 v1:172.21.15.196:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f7678002100 con 0x7f76800b0910 2024-03-29T10:40:41.208 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.201+0000 7f7676ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:41.208 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.201+0000 7f7676ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:40:41.208 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.201+0000 7f7676ffd640 10 monclient: dump: 2024-03-29T10:40:41.208 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 1 2024-03-29T10:40:41.208 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:40:41.208 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:40:41.208 INFO:teuthology.orchestra.run.smithi196.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:40:41.208 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:41.208 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 1 2024-03-29T10:40:41.208 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:40:41.208 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:40:41.208 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:40:41.209 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-29T10:40:41.209 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.201+0000 7f7684d53640 4 set_mon_vals no callback set 2024-03-29T10:40:41.209 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.201+0000 7f7676ffd640 1 -- 172.21.15.196:0/962343532 <== mon.1 v1:172.21.15.196:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f7678003f90 con 0x7f76800b0910 2024-03-29T10:40:41.209 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.201+0000 7f7676ffd640 1 -- 172.21.15.196:0/962343532 <== mon.1 v1:172.21.15.196:6789/0 8 ==== osd_map(3..3 src has 1..3) v4 ==== 840+0+0 (unknown 3339249578 0 0) 0x7f7678006690 con 0x7f76800b0910 2024-03-29T10:40:41.209 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.201+0000 7f7676ffd640 10 client.4117.objecter ms_dispatch 0x7f7680000b90 osd_map(3..3 src has 1..3) v4 2024-03-29T10:40:41.209 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.201+0000 7f7676ffd640 3 client.4117.objecter handle_osd_map got epochs [3,3] > 0 2024-03-29T10:40:41.209 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.201+0000 7f7676ffd640 3 client.4117.objecter handle_osd_map decoding full epoch 3 2024-03-29T10:40:41.209 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.201+0000 7f7676ffd640 20 client.4117.objecter dump_active .. 0 homeless 2024-03-29T10:40:41.209 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.202+0000 7f76887ba640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:40:41.209 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.202+0000 7f76887ba640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:40:41.209 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.202+0000 7f76887ba640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T10:40:41.209 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.202+0000 7f76887ba640 1 -- 172.21.15.196:0/962343532 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f768009b020 con 0x7f76800b0910 2024-03-29T10:40:41.209 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.204+0000 7f7676ffd640 1 -- 172.21.15.196:0/962343532 <== mon.1 v1:172.21.15.196:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f7678005b40 con 0x7f76800b0910 2024-03-29T10:40:41.209 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.204+0000 7f7676ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:40:41.210 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.204+0000 7f7676ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:40:41.260 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.276+0000 7f76887ba640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "5de24915-8dec-49b6-8b16-f3170197c591", "id": 1}] 2024-03-29T10:40:41.260 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.276+0000 7f76887ba640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "5de24915-8dec-49b6-8b16-f3170197c591", "id": 1}] 2024-03-29T10:40:41.260 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.276+0000 7f76887ba640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T10:40:41.260 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.276+0000 7f76887ba640 1 -- 172.21.15.196:0/962343532 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "osd new", "uuid": "5de24915-8dec-49b6-8b16-f3170197c591", "id": 1} v 0) v1 -- 0x7f768004f730 con 0x7f76800b0910 2024-03-29T10:40:41.265 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.281+0000 7f7676ffd640 1 -- 172.21.15.196:0/962343532 <== mon.1 v1:172.21.15.196:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "5de24915-8dec-49b6-8b16-f3170197c591", "id": 1}]=0 v4) v1 ==== 112+0+2 (unknown 2270357742 0 940483276) 0x7f7678004240 con 0x7f76800b0910 2024-03-29T10:40:41.265 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.281+0000 7f7676ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "5de24915-8dec-49b6-8b16-f3170197c591", "id": 1}] 2024-03-29T10:40:41.265 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.281+0000 7f7676ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:40:41.265 INFO:teuthology.orchestra.run.smithi196.stdout:1 2024-03-29T10:40:41.266 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.282+0000 7f76887ba640 10 librados: watch_flush enter 2024-03-29T10:40:41.266 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.282+0000 7f76887ba640 10 librados: watch_flush exit 2024-03-29T10:40:41.266 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.282+0000 7f76887ba640 20 client.4117.objecter shutdown clearing up homeless session... 2024-03-29T10:40:41.267 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.282+0000 7f76887ba640 10 client.4117.objecter successfully canceled tick 2024-03-29T10:40:41.267 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.282+0000 7f76887ba640 10 monclient: shutdown 2024-03-29T10:40:41.267 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.282+0000 7f76887ba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:40:41.267 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.282+0000 7f76887ba640 1 -- 172.21.15.196:0/962343532 >> v1:172.21.15.196:6789/0 conn(0x7f76800b0910 legacy=0x7f76801e1b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:41.267 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.282+0000 7f76887ba640 1 -- 172.21.15.196:0/962343532 shutdown_connections 2024-03-29T10:40:41.267 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.282+0000 7f76887ba640 1 -- 172.21.15.196:0/962343532 >> 172.21.15.196:0/962343532 conn(0x7f7680066b00 msgr2=0x7f76800b4a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:40:41.267 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.282+0000 7f76887ba640 1 -- 172.21.15.196:0/962343532 shutdown_connections 2024-03-29T10:40:41.267 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.282+0000 7f76887ba640 1 -- 172.21.15.196:0/962343532 wait complete. 2024-03-29T10:40:41.267 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.282+0000 7f76887ba640 1 librados: shutdown 2024-03-29T10:40:41.276 DEBUG:teuthology.orchestra.run.smithi196:> sudo ceph --cluster ceph osd new ac054af3-d589-4565-9644-e6af75975c53 2 2024-03-29T10:40:41.375 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.392+0000 7f0a7fd82640 10 monclient: get_monmap_and_config 2024-03-29T10:40:41.376 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.392+0000 7f0a7fd82640 10 monclient: build_initial_monmap 2024-03-29T10:40:41.376 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.392+0000 7f0a7fd82640 10 monclient: monmap: 2024-03-29T10:40:41.376 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 0 2024-03-29T10:40:41.376 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:40:41.376 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 2024-03-29T10:40:41.393292+0000 2024-03-29T10:40:41.376 INFO:teuthology.orchestra.run.smithi196.stderr:created 2024-03-29T10:40:41.393292+0000 2024-03-29T10:40:41.376 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 0 (unknown) 2024-03-29T10:40:41.376 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 1 2024-03-29T10:40:41.376 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-03-29T10:40:41.376 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-29T10:40:41.376 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.88:6790/0 mon.noname-b 2024-03-29T10:40:41.376 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-29T10:40:41.380 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.392+0000 7f0a7fd82640 1 Processor -- start 2024-03-29T10:40:41.380 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.393+0000 7f0a7fd82640 1 -- start start 2024-03-29T10:40:41.380 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.393+0000 7f0a7fd82640 10 monclient: init 2024-03-29T10:40:41.380 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.393+0000 7f0a7fd82640 10 monclient: _reopen_session rank -1 2024-03-29T10:40:41.380 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.393+0000 7f0a7fd82640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:40:41.380 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.393+0000 7f0a7fd82640 10 monclient(hunting): picked mon.noname-c con 0x7f0a780b1780 addr v1:172.21.15.196:6789/0 2024-03-29T10:40:41.380 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.393+0000 7f0a7fd82640 10 monclient(hunting): picked mon.noname-b con 0x7f0a7813d7e0 addr v1:172.21.15.88:6790/0 2024-03-29T10:40:41.381 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.393+0000 7f0a7fd82640 10 monclient(hunting): picked mon.noname-a con 0x7f0a78143be0 addr v1:172.21.15.88:6789/0 2024-03-29T10:40:41.381 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.393+0000 7f0a7fd82640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a78070b30 con 0x7f0a78143be0 2024-03-29T10:40:41.381 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.393+0000 7f0a7fd82640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a78070da0 con 0x7f0a780b1780 2024-03-29T10:40:41.381 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.393+0000 7f0a7fd82640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a7809adb0 con 0x7f0a7813d7e0 2024-03-29T10:40:41.381 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.393+0000 7f0a7fd82640 10 monclient(hunting): _renew_subs 2024-03-29T10:40:41.381 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.393+0000 7f0a7fd82640 10 monclient(hunting): authenticate will time out at 787.708835s 2024-03-29T10:40:41.381 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.393+0000 7f0a7db1e640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f0a780b1780 0x7f0a7813d0d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.196:39906/0 (socket says 172.21.15.196:39906) 2024-03-29T10:40:41.381 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.393+0000 7f0a7db1e640 1 -- 172.21.15.196:0/1127441748 learned_addr learned my addr 172.21.15.196:0/1127441748 (peer_addr_for_me v1:172.21.15.196:0/0) 2024-03-29T10:40:41.381 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.394+0000 7f0a7cb1c640 1 -- 172.21.15.196:0/1127441748 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 415118165 0 0) 0x7f0a78070da0 con 0x7f0a780b1780 2024-03-29T10:40:41.381 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.394+0000 7f0a7cb1c640 10 monclient(hunting): _negotiate 2024-03-29T10:40:41.381 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.394+0000 7f0a7cb1c640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:41.381 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.394+0000 7f0a7cb1c640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:41.381 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.394+0000 7f0a7cb1c640 10 monclient(hunting): my global_id is 4120 2024-03-29T10:40:41.382 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.394+0000 7f0a7cb1c640 1 -- 172.21.15.196:0/1127441748 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0a64003520 con 0x7f0a780b1780 2024-03-29T10:40:41.382 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.394+0000 7f0a7cb1c640 1 -- 172.21.15.196:0/1127441748 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3637804357 0 0) 0x7f0a7809adb0 con 0x7f0a7813d7e0 2024-03-29T10:40:41.382 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.394+0000 7f0a7cb1c640 10 monclient(hunting): _negotiate 2024-03-29T10:40:41.382 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.394+0000 7f0a7cb1c640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:41.382 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.394+0000 7f0a7cb1c640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:41.382 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.394+0000 7f0a7cb1c640 10 monclient(hunting): my global_id is 4118 2024-03-29T10:40:41.382 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.394+0000 7f0a7cb1c640 1 -- 172.21.15.196:0/1127441748 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0a78070da0 con 0x7f0a7813d7e0 2024-03-29T10:40:41.382 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.394+0000 7f0a7cb1c640 1 -- 172.21.15.196:0/1127441748 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3363562779 0 0) 0x7f0a64003520 con 0x7f0a780b1780 2024-03-29T10:40:41.382 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.394+0000 7f0a7cb1c640 1 -- 172.21.15.196:0/1127441748 >> v1:172.21.15.88:6790/0 conn(0x7f0a7813d7e0 legacy=0x7f0a781434d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:41.382 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.394+0000 7f0a7cb1c640 1 -- 172.21.15.196:0/1127441748 >> v1:172.21.15.88:6789/0 conn(0x7f0a78143be0 legacy=0x7f0a7814b780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:41.382 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.394+0000 7f0a7cb1c640 10 monclient: _finish_hunting 0 2024-03-29T10:40:41.382 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.394+0000 7f0a7cb1c640 1 monclient: found mon.noname-c 2024-03-29T10:40:41.382 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.394+0000 7f0a7cb1c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:40:41.382 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.394+0000 7f0a7cb1c640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.196:6789/0 2024-03-29T10:40:41.383 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.394+0000 7f0a7cb1c640 1 -- 172.21.15.196:0/1127441748 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a78144070 con 0x7f0a780b1780 2024-03-29T10:40:41.383 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.394+0000 7f0a7cb1c640 10 monclient: _finish_auth 0 2024-03-29T10:40:41.383 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.394+0000 7f0a7cb1c640 10 monclient: _check_auth_tickets 2024-03-29T10:40:41.383 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.394+0000 7f0a7cb1c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:40:41.383 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.394+0000 7f0a7cb1c640 1 -- 172.21.15.196:0/1127441748 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f0a74001690 con 0x7f0a780b1780 2024-03-29T10:40:41.383 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.394+0000 7f0a7fd82640 5 monclient: authenticate success, global_id 4120 2024-03-29T10:40:41.383 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.394+0000 7f0a7fd82640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:40:41.383 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.394+0000 7f0a7cb1c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:41.383 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.394+0000 7f0a7cb1c640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:40:41.383 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.394+0000 7f0a7cb1c640 10 monclient: dump: 2024-03-29T10:40:41.383 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 1 2024-03-29T10:40:41.383 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:40:41.383 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:40:41.383 INFO:teuthology.orchestra.run.smithi196.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:40:41.383 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:41.384 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 1 2024-03-29T10:40:41.384 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:40:41.384 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:40:41.384 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:40:41.384 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-29T10:40:41.384 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.394+0000 7f0a7fd82640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:40:41.384 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.394+0000 7f0a7cb1c640 1 -- 172.21.15.196:0/1127441748 <== mon.1 v1:172.21.15.196:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0a74001870 con 0x7f0a780b1780 2024-03-29T10:40:41.384 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.394+0000 7f0a7cb1c640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:40:41.384 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.394+0000 7f0a7cb1c640 1 -- 172.21.15.196:0/1127441748 <== mon.1 v1:172.21.15.196:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f0a74001c80 con 0x7f0a780b1780 2024-03-29T10:40:41.384 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.394+0000 7f0a7cb1c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:41.384 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.394+0000 7f0a7cb1c640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:40:41.384 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.394+0000 7f0a7cb1c640 10 monclient: dump: 2024-03-29T10:40:41.384 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 1 2024-03-29T10:40:41.384 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:40:41.385 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:40:41.385 INFO:teuthology.orchestra.run.smithi196.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:40:41.385 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:41.385 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 1 2024-03-29T10:40:41.385 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:40:41.385 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:40:41.385 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:40:41.385 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-29T10:40:41.385 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.394+0000 7f0a7fd82640 10 monclient: get_monmap_and_config success 2024-03-29T10:40:41.385 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.394+0000 7f0a7fd82640 4 set_mon_vals no callback set 2024-03-29T10:40:41.385 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.395+0000 7f0a7fd82640 10 monclient: shutdown 2024-03-29T10:40:41.385 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.395+0000 7f0a7fd82640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:40:41.385 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.395+0000 7f0a7fd82640 1 -- 172.21.15.196:0/1127441748 >> v1:172.21.15.196:6789/0 conn(0x7f0a780b1780 legacy=0x7f0a7813d0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:41.385 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.395+0000 7f0a7fd82640 1 -- 172.21.15.196:0/1127441748 shutdown_connections 2024-03-29T10:40:41.385 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.395+0000 7f0a7fd82640 1 -- 172.21.15.196:0/1127441748 >> 172.21.15.196:0/1127441748 conn(0x7f0a780b58f0 msgr2=0x7f0a780b5d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:40:41.386 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.395+0000 7f0a7fd82640 1 -- 172.21.15.196:0/1127441748 shutdown_connections 2024-03-29T10:40:41.386 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.395+0000 7f0a7fd82640 1 -- 172.21.15.196:0/1127441748 wait complete. 2024-03-29T10:40:41.386 INFO:tasks.ceph.mgr.x.smithi196.stderr:2024-03-29T10:40:41.395+0000 7ff5dcc4b040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-29T10:40:41.391 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.395+0000 7f0a7fd82640 10 monclient: build_initial_monmap 2024-03-29T10:40:41.392 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.395+0000 7f0a7fd82640 10 monclient: monmap: 2024-03-29T10:40:41.392 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 0 2024-03-29T10:40:41.392 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:40:41.392 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 0.000000 2024-03-29T10:40:41.392 INFO:teuthology.orchestra.run.smithi196.stderr:created 0.000000 2024-03-29T10:40:41.392 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 0 (unknown) 2024-03-29T10:40:41.392 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 1 2024-03-29T10:40:41.392 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-03-29T10:40:41.392 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-29T10:40:41.392 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.88:6790/0 mon.noname-c 2024-03-29T10:40:41.392 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-29T10:40:41.392 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.396+0000 7f0a7fd82640 1 librados: starting msgr at 2024-03-29T10:40:41.392 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.396+0000 7f0a7fd82640 1 librados: starting objecter 2024-03-29T10:40:41.392 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.396+0000 7f0a7fd82640 1 Processor -- start 2024-03-29T10:40:41.393 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.396+0000 7f0a7fd82640 1 -- start start 2024-03-29T10:40:41.393 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.396+0000 7f0a7fd82640 1 librados: setting wanted keys 2024-03-29T10:40:41.393 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.396+0000 7f0a7fd82640 1 librados: calling monclient init 2024-03-29T10:40:41.393 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.396+0000 7f0a7fd82640 10 monclient: init 2024-03-29T10:40:41.393 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.396+0000 7f0a7fd82640 10 monclient: _reopen_session rank -1 2024-03-29T10:40:41.393 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.396+0000 7f0a7fd82640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:40:41.393 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.396+0000 7f0a7fd82640 10 monclient(hunting): picked mon.noname-a con 0x7f0a780b1780 addr v1:172.21.15.88:6789/0 2024-03-29T10:40:41.393 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.396+0000 7f0a7fd82640 10 monclient(hunting): picked mon.noname-b con 0x7f0a7813d7e0 addr v1:172.21.15.196:6789/0 2024-03-29T10:40:41.393 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.396+0000 7f0a7fd82640 10 monclient(hunting): picked mon.noname-c con 0x7f0a78143be0 addr v1:172.21.15.88:6790/0 2024-03-29T10:40:41.393 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.396+0000 7f0a7fd82640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a7806c080 con 0x7f0a780b1780 2024-03-29T10:40:41.393 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.396+0000 7f0a7fd82640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a7806ac40 con 0x7f0a7813d7e0 2024-03-29T10:40:41.393 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.396+0000 7f0a7fd82640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a7806bcc0 con 0x7f0a78143be0 2024-03-29T10:40:41.393 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.396+0000 7f0a7fd82640 10 monclient(hunting): _renew_subs 2024-03-29T10:40:41.393 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.396+0000 7f0a7fd82640 10 monclient(hunting): authenticate will time out at 787.711803s 2024-03-29T10:40:41.394 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.396+0000 7f0a7d31d640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f0a7813d7e0 0x7f0a7806b5b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.196:39912/0 (socket says 172.21.15.196:39912) 2024-03-29T10:40:41.394 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.396+0000 7f0a7d31d640 1 -- 172.21.15.196:0/2919659905 learned_addr learned my addr 172.21.15.196:0/2919659905 (peer_addr_for_me v1:172.21.15.196:0/0) 2024-03-29T10:40:41.394 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.396+0000 7f0a6e7fc640 10 client.?.objecter ms_handle_connect 0x7f0a7813d7e0 2024-03-29T10:40:41.394 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.396+0000 7f0a6e7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:40:41.394 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.397+0000 7f0a6e7fc640 10 client.?.objecter ms_handle_connect 0x7f0a78143be0 2024-03-29T10:40:41.394 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.397+0000 7f0a6e7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:40:41.394 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.397+0000 7f0a6e7fc640 10 client.?.objecter ms_handle_connect 0x7f0a780b1780 2024-03-29T10:40:41.394 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.397+0000 7f0a6e7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:40:41.394 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.397+0000 7f0a6e7fc640 1 -- 172.21.15.196:0/2919659905 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2404187300 0 0) 0x7f0a7806ac40 con 0x7f0a7813d7e0 2024-03-29T10:40:41.394 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.397+0000 7f0a6e7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:40:41.394 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.397+0000 7f0a6e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:41.394 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.397+0000 7f0a6e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:41.394 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.397+0000 7f0a6e7fc640 10 monclient(hunting): my global_id is 4123 2024-03-29T10:40:41.394 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.397+0000 7f0a6e7fc640 1 -- 172.21.15.196:0/2919659905 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0a5c004eb0 con 0x7f0a7813d7e0 2024-03-29T10:40:41.395 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.397+0000 7f0a6e7fc640 1 -- 172.21.15.196:0/2919659905 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2117888756 0 0) 0x7f0a7806bcc0 con 0x7f0a78143be0 2024-03-29T10:40:41.395 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.397+0000 7f0a6e7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:40:41.395 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.397+0000 7f0a6e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:41.395 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.397+0000 7f0a6e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:41.395 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.397+0000 7f0a6e7fc640 10 monclient(hunting): my global_id is 4121 2024-03-29T10:40:41.395 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.397+0000 7f0a6e7fc640 1 -- 172.21.15.196:0/2919659905 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0a7806ac40 con 0x7f0a78143be0 2024-03-29T10:40:41.395 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.397+0000 7f0a6e7fc640 1 -- 172.21.15.196:0/2919659905 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2938858242 0 0) 0x7f0a7806c080 con 0x7f0a780b1780 2024-03-29T10:40:41.395 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.397+0000 7f0a6e7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:40:41.395 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.397+0000 7f0a6e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:41.395 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.397+0000 7f0a6e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:41.395 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.397+0000 7f0a6e7fc640 10 monclient(hunting): my global_id is 4125 2024-03-29T10:40:41.395 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.397+0000 7f0a6e7fc640 1 -- 172.21.15.196:0/2919659905 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0a7806bcc0 con 0x7f0a780b1780 2024-03-29T10:40:41.395 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.397+0000 7f0a6e7fc640 1 -- 172.21.15.196:0/2919659905 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 8979146 0 0) 0x7f0a5c004eb0 con 0x7f0a7813d7e0 2024-03-29T10:40:41.395 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.397+0000 7f0a6e7fc640 1 -- 172.21.15.196:0/2919659905 --> v1:172.21.15.196:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0a7806c080 con 0x7f0a7813d7e0 2024-03-29T10:40:41.396 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.397+0000 7f0a6e7fc640 1 -- 172.21.15.196:0/2919659905 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f0a68001a10 con 0x7f0a7813d7e0 2024-03-29T10:40:41.396 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.397+0000 7f0a6e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:41.396 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.397+0000 7f0a6e7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:40:41.396 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.397+0000 7f0a6e7fc640 10 monclient(hunting): dump: 2024-03-29T10:40:41.396 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 1 2024-03-29T10:40:41.396 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:40:41.396 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:40:41.396 INFO:teuthology.orchestra.run.smithi196.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:40:41.396 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:41.396 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 1 2024-03-29T10:40:41.396 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:40:41.396 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:40:41.396 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:40:41.396 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-29T10:40:41.396 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.397+0000 7f0a6e7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:40:41.397 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.397+0000 7f0a6e7fc640 1 -- 172.21.15.196:0/2919659905 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3995173680 0 0) 0x7f0a7806bcc0 con 0x7f0a780b1780 2024-03-29T10:40:41.397 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.397+0000 7f0a6e7fc640 1 -- 172.21.15.196:0/2919659905 --> v1:172.21.15.88:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0a5c004eb0 con 0x7f0a780b1780 2024-03-29T10:40:41.397 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.397+0000 7f0a6e7fc640 1 -- 172.21.15.196:0/2919659905 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 67230686 0 0) 0x7f0a7806ac40 con 0x7f0a78143be0 2024-03-29T10:40:41.397 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.397+0000 7f0a6e7fc640 1 -- 172.21.15.196:0/2919659905 --> v1:172.21.15.88:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0a7806bcc0 con 0x7f0a78143be0 2024-03-29T10:40:41.397 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.397+0000 7f0a6e7fc640 1 -- 172.21.15.196:0/2919659905 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f0a74001b80 con 0x7f0a780b1780 2024-03-29T10:40:41.397 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.397+0000 7f0a6e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:41.397 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.397+0000 7f0a6e7fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:40:41.397 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.397+0000 7f0a6e7fc640 10 monclient(hunting): dump: 2024-03-29T10:40:41.397 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 1 2024-03-29T10:40:41.397 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:40:41.397 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:40:41.397 INFO:teuthology.orchestra.run.smithi196.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:40:41.397 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:41.397 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 1 2024-03-29T10:40:41.397 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:40:41.398 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:40:41.398 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:40:41.398 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-29T10:40:41.398 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.397+0000 7f0a6e7fc640 1 -- 172.21.15.196:0/2919659905 <== mon.1 v1:172.21.15.196:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3747292771 0 0) 0x7f0a7806c080 con 0x7f0a7813d7e0 2024-03-29T10:40:41.398 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.397+0000 7f0a6e7fc640 1 -- 172.21.15.196:0/2919659905 >> v1:172.21.15.88:6790/0 conn(0x7f0a78143be0 legacy=0x7f0a781e1d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:41.398 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.397+0000 7f0a6e7fc640 1 -- 172.21.15.196:0/2919659905 >> v1:172.21.15.88:6789/0 conn(0x7f0a780b1780 legacy=0x7f0a7806c510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:41.398 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.397+0000 7f0a6e7fc640 10 monclient: _finish_hunting 0 2024-03-29T10:40:41.398 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.397+0000 7f0a6e7fc640 1 monclient: found mon.b 2024-03-29T10:40:41.398 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.397+0000 7f0a6e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:40:41.398 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.397+0000 7f0a6e7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T10:40:41.398 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.397+0000 7f0a6e7fc640 1 -- 172.21.15.196:0/2919659905 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a780af360 con 0x7f0a7813d7e0 2024-03-29T10:40:41.398 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.397+0000 7f0a6e7fc640 10 monclient: _finish_auth 0 2024-03-29T10:40:41.398 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.397+0000 7f0a6e7fc640 10 monclient: _check_auth_tickets 2024-03-29T10:40:41.399 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.397+0000 7f0a6e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:40:41.399 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.397+0000 7f0a7fd82640 5 monclient: authenticate success, global_id 4123 2024-03-29T10:40:41.399 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.397+0000 7f0a7fd82640 10 monclient: _renew_subs 2024-03-29T10:40:41.399 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.397+0000 7f0a7fd82640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T10:40:41.399 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.397+0000 7f0a7fd82640 1 -- 172.21.15.196:0/2919659905 --> v1:172.21.15.196:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a780af5b0 con 0x7f0a7813d7e0 2024-03-29T10:40:41.399 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.397+0000 7f0a7fd82640 10 client.4123.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:40:41.399 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.397+0000 7f0a7fd82640 10 monclient: _renew_subs 2024-03-29T10:40:41.399 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.397+0000 7f0a7fd82640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T10:40:41.399 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.397+0000 7f0a7fd82640 1 -- 172.21.15.196:0/2919659905 --> v1:172.21.15.196:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a781da970 con 0x7f0a7813d7e0 2024-03-29T10:40:41.399 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.397+0000 7f0a7fd82640 1 librados: init done 2024-03-29T10:40:41.399 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.397+0000 7f0a6e7fc640 1 -- 172.21.15.196:0/2919659905 <== mon.1 v1:172.21.15.196:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0a68001c50 con 0x7f0a7813d7e0 2024-03-29T10:40:41.399 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.397+0000 7f0a6e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:40:41.399 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.397+0000 7f0a6e7fc640 1 -- 172.21.15.196:0/2919659905 <== mon.1 v1:172.21.15.196:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f0a68003a60 con 0x7f0a7813d7e0 2024-03-29T10:40:41.399 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.397+0000 7f0a6e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:41.399 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.397+0000 7f0a6e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:40:41.400 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.397+0000 7f0a6e7fc640 10 monclient: dump: 2024-03-29T10:40:41.400 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 1 2024-03-29T10:40:41.400 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:40:41.400 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:40:41.400 INFO:teuthology.orchestra.run.smithi196.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:40:41.400 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:41.400 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 1 2024-03-29T10:40:41.400 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:40:41.400 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:40:41.400 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:40:41.400 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-29T10:40:41.400 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.397+0000 7f0a6ffff640 4 set_mon_vals no callback set 2024-03-29T10:40:41.400 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.397+0000 7f0a6e7fc640 1 -- 172.21.15.196:0/2919659905 <== mon.1 v1:172.21.15.196:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f0a68003fd0 con 0x7f0a7813d7e0 2024-03-29T10:40:41.400 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.397+0000 7f0a6e7fc640 1 -- 172.21.15.196:0/2919659905 <== mon.1 v1:172.21.15.196:6789/0 8 ==== osd_map(4..4 src has 1..4) v4 ==== 955+0+0 (unknown 2738672693 0 0) 0x7f0a68006710 con 0x7f0a7813d7e0 2024-03-29T10:40:41.400 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.397+0000 7f0a6e7fc640 10 client.4123.objecter ms_dispatch 0x7f0a78000b90 osd_map(4..4 src has 1..4) v4 2024-03-29T10:40:41.401 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.397+0000 7f0a6e7fc640 3 client.4123.objecter handle_osd_map got epochs [4,4] > 0 2024-03-29T10:40:41.401 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.397+0000 7f0a6e7fc640 3 client.4123.objecter handle_osd_map decoding full epoch 4 2024-03-29T10:40:41.401 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.397+0000 7f0a6e7fc640 20 client.4123.objecter dump_active .. 0 homeless 2024-03-29T10:40:41.401 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.398+0000 7f0a7fd82640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:40:41.401 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.398+0000 7f0a7fd82640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:40:41.401 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.398+0000 7f0a7fd82640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T10:40:41.401 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.398+0000 7f0a7fd82640 1 -- 172.21.15.196:0/2919659905 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a60005080 con 0x7f0a7813d7e0 2024-03-29T10:40:41.401 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.400+0000 7f0a6e7fc640 1 -- 172.21.15.196:0/2919659905 <== mon.1 v1:172.21.15.196:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f0a68004460 con 0x7f0a7813d7e0 2024-03-29T10:40:41.401 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.400+0000 7f0a6e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:40:41.401 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.400+0000 7f0a6e7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:40:41.402 INFO:tasks.ceph.mgr.y.smithi088.stderr:2024-03-29T10:40:41.412+0000 7fca56440040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-29T10:40:41.443 INFO:tasks.ceph.mgr.x.smithi196.stderr:2024-03-29T10:40:41.460+0000 7ff5dcc4b040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-29T10:40:41.449 INFO:tasks.ceph.mgr.y.smithi088.stderr:2024-03-29T10:40:41.466+0000 7fca56440040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-29T10:40:41.466 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.482+0000 7f0a7fd82640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "ac054af3-d589-4565-9644-e6af75975c53", "id": 2}] 2024-03-29T10:40:41.466 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.483+0000 7f0a7fd82640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "ac054af3-d589-4565-9644-e6af75975c53", "id": 2}] 2024-03-29T10:40:41.466 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.483+0000 7f0a7fd82640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T10:40:41.466 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.483+0000 7f0a7fd82640 1 -- 172.21.15.196:0/2919659905 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "osd new", "uuid": "ac054af3-d589-4565-9644-e6af75975c53", "id": 2} v 0) v1 -- 0x7f0a600058b0 con 0x7f0a7813d7e0 2024-03-29T10:40:41.470 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.486+0000 7f0a6e7fc640 1 -- 172.21.15.196:0/2919659905 <== mon.1 v1:172.21.15.196:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "ac054af3-d589-4565-9644-e6af75975c53", "id": 2}]=0 v5) v1 ==== 112+0+2 (unknown 633246252 0 216599125) 0x7f0a68006400 con 0x7f0a7813d7e0 2024-03-29T10:40:41.470 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.487+0000 7f0a6e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "ac054af3-d589-4565-9644-e6af75975c53", "id": 2}] 2024-03-29T10:40:41.470 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.487+0000 7f0a6e7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:40:41.470 INFO:teuthology.orchestra.run.smithi196.stdout:2 2024-03-29T10:40:41.472 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.487+0000 7f0a7fd82640 10 librados: watch_flush enter 2024-03-29T10:40:41.472 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.487+0000 7f0a7fd82640 10 librados: watch_flush exit 2024-03-29T10:40:41.472 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.487+0000 7f0a7fd82640 20 client.4123.objecter shutdown clearing up homeless session... 2024-03-29T10:40:41.472 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.487+0000 7f0a7fd82640 10 client.4123.objecter successfully canceled tick 2024-03-29T10:40:41.472 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.487+0000 7f0a7fd82640 10 monclient: shutdown 2024-03-29T10:40:41.472 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.487+0000 7f0a7fd82640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:40:41.472 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.487+0000 7f0a7fd82640 1 -- 172.21.15.196:0/2919659905 >> v1:172.21.15.196:6789/0 conn(0x7f0a7813d7e0 legacy=0x7f0a7806b5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:41.472 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.487+0000 7f0a7fd82640 1 -- 172.21.15.196:0/2919659905 shutdown_connections 2024-03-29T10:40:41.472 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.487+0000 7f0a7fd82640 1 -- 172.21.15.196:0/2919659905 >> 172.21.15.196:0/2919659905 conn(0x7f0a780b58f0 msgr2=0x7f0a78069bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:40:41.472 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.487+0000 7f0a7fd82640 1 -- 172.21.15.196:0/2919659905 shutdown_connections 2024-03-29T10:40:41.472 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.487+0000 7f0a7fd82640 1 -- 172.21.15.196:0/2919659905 wait complete. 2024-03-29T10:40:41.472 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.488+0000 7f0a7fd82640 1 librados: shutdown 2024-03-29T10:40:41.481 DEBUG:teuthology.orchestra.run.smithi196:> sudo ceph --cluster ceph osd new 3ec4f045-c714-4a22-bd1a-e07330b45ab3 3 2024-03-29T10:40:41.491 INFO:tasks.ceph.mgr.x.smithi196.stderr:2024-03-29T10:40:41.507+0000 7ff5dcc4b040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-29T10:40:41.494 INFO:tasks.ceph.mgr.y.smithi088.stderr:2024-03-29T10:40:41.511+0000 7fca56440040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-29T10:40:41.548 INFO:tasks.ceph.mgr.x.smithi196.stderr:2024-03-29T10:40:41.564+0000 7ff5dcc4b040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-29T10:40:41.548 INFO:tasks.ceph.mgr.y.smithi088.stderr:2024-03-29T10:40:41.565+0000 7fca56440040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-29T10:40:41.580 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.597+0000 7f44d9633640 10 monclient: get_monmap_and_config 2024-03-29T10:40:41.580 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.597+0000 7f44d9633640 10 monclient: build_initial_monmap 2024-03-29T10:40:41.581 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.597+0000 7f44d9633640 10 monclient: monmap: 2024-03-29T10:40:41.581 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 0 2024-03-29T10:40:41.581 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:40:41.581 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 2024-03-29T10:40:41.597997+0000 2024-03-29T10:40:41.581 INFO:teuthology.orchestra.run.smithi196.stderr:created 2024-03-29T10:40:41.597997+0000 2024-03-29T10:40:41.581 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 0 (unknown) 2024-03-29T10:40:41.581 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 1 2024-03-29T10:40:41.581 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-03-29T10:40:41.581 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-29T10:40:41.581 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.88:6790/0 mon.noname-b 2024-03-29T10:40:41.581 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-29T10:40:41.590 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.597+0000 7f44d9633640 1 Processor -- start 2024-03-29T10:40:41.590 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.597+0000 7f44d9633640 1 -- start start 2024-03-29T10:40:41.590 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.597+0000 7f44d9633640 10 monclient: init 2024-03-29T10:40:41.590 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.598+0000 7f44d9633640 10 monclient: _reopen_session rank -1 2024-03-29T10:40:41.590 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.598+0000 7f44d9633640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:40:41.590 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.598+0000 7f44d9633640 10 monclient(hunting): picked mon.noname-b con 0x7f44d409ac40 addr v1:172.21.15.88:6790/0 2024-03-29T10:40:41.590 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.598+0000 7f44d9633640 10 monclient(hunting): picked mon.noname-a con 0x7f44d409c870 addr v1:172.21.15.88:6789/0 2024-03-29T10:40:41.591 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.598+0000 7f44d9633640 10 monclient(hunting): picked mon.noname-c con 0x7f44d40b4360 addr v1:172.21.15.196:6789/0 2024-03-29T10:40:41.591 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.598+0000 7f44d9633640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44d409dec0 con 0x7f44d409c870 2024-03-29T10:40:41.591 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.598+0000 7f44d9633640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44d409da30 con 0x7f44d40b4360 2024-03-29T10:40:41.591 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.598+0000 7f44d9633640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44d40b39c0 con 0x7f44d409ac40 2024-03-29T10:40:41.591 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.598+0000 7f44d9633640 10 monclient(hunting): _renew_subs 2024-03-29T10:40:41.591 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.598+0000 7f44d9633640 10 monclient(hunting): authenticate will time out at 787.913374s 2024-03-29T10:40:41.591 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.598+0000 7f44d37fe640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f44d40b4360 0x7f44d414d9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.196:39914/0 (socket says 172.21.15.196:39914) 2024-03-29T10:40:41.591 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.598+0000 7f44d37fe640 1 -- 172.21.15.196:0/4069837445 learned_addr learned my addr 172.21.15.196:0/4069837445 (peer_addr_for_me v1:172.21.15.196:0/0) 2024-03-29T10:40:41.591 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.598+0000 7f44d1ffb640 1 -- 172.21.15.196:0/4069837445 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3347371373 0 0) 0x7f44d409da30 con 0x7f44d40b4360 2024-03-29T10:40:41.591 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.598+0000 7f44d1ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:40:41.591 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.598+0000 7f44d1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:41.591 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.598+0000 7f44d1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:41.591 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.598+0000 7f44d1ffb640 10 monclient(hunting): my global_id is 4126 2024-03-29T10:40:41.592 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.598+0000 7f44d1ffb640 1 -- 172.21.15.196:0/4069837445 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f44bc003520 con 0x7f44d40b4360 2024-03-29T10:40:41.592 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.598+0000 7f44d1ffb640 1 -- 172.21.15.196:0/4069837445 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4165216558 0 0) 0x7f44c4001110 con 0x7f44d409c870 2024-03-29T10:40:41.592 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.598+0000 7f44d1ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:40:41.592 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.598+0000 7f44d1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:41.592 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.598+0000 7f44d1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:41.592 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.598+0000 7f44d1ffb640 10 monclient(hunting): my global_id is 4128 2024-03-29T10:40:41.592 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.598+0000 7f44d1ffb640 1 -- 172.21.15.196:0/4069837445 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f44d409da30 con 0x7f44d409c870 2024-03-29T10:40:41.592 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.598+0000 7f44d1ffb640 1 -- 172.21.15.196:0/4069837445 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2173924955 0 0) 0x7f44d40b39c0 con 0x7f44d409ac40 2024-03-29T10:40:41.592 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.598+0000 7f44d1ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:40:41.592 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.598+0000 7f44d1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:41.592 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.598+0000 7f44d1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:41.592 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.598+0000 7f44d1ffb640 10 monclient(hunting): my global_id is 4124 2024-03-29T10:40:41.592 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.598+0000 7f44d1ffb640 1 -- 172.21.15.196:0/4069837445 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f44c4001110 con 0x7f44d409ac40 2024-03-29T10:40:41.592 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.599+0000 7f44d1ffb640 1 -- 172.21.15.196:0/4069837445 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2808033502 0 0) 0x7f44bc003520 con 0x7f44d40b4360 2024-03-29T10:40:41.592 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.599+0000 7f44d1ffb640 1 -- 172.21.15.196:0/4069837445 >> v1:172.21.15.88:6790/0 conn(0x7f44d409ac40 legacy=0x7f44d40b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:41.593 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.599+0000 7f44d1ffb640 1 -- 172.21.15.196:0/4069837445 >> v1:172.21.15.88:6789/0 conn(0x7f44d409c870 legacy=0x7f44d413f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:41.593 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.599+0000 7f44d1ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:40:41.593 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.599+0000 7f44d1ffb640 1 monclient: found mon.noname-c 2024-03-29T10:40:41.593 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.599+0000 7f44d1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:40:41.593 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.599+0000 7f44d1ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.196:6789/0 2024-03-29T10:40:41.593 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.599+0000 7f44d1ffb640 1 -- 172.21.15.196:0/4069837445 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44d40b5160 con 0x7f44d40b4360 2024-03-29T10:40:41.593 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.599+0000 7f44d1ffb640 10 monclient: _finish_auth 0 2024-03-29T10:40:41.593 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.599+0000 7f44d1ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:40:41.593 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.599+0000 7f44d1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:40:41.593 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.599+0000 7f44d1ffb640 1 -- 172.21.15.196:0/4069837445 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f44cc001a10 con 0x7f44d40b4360 2024-03-29T10:40:41.593 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.599+0000 7f44d9633640 5 monclient: authenticate success, global_id 4126 2024-03-29T10:40:41.593 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.599+0000 7f44d9633640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:40:41.593 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.599+0000 7f44d1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:41.593 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.599+0000 7f44d1ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:40:41.594 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.599+0000 7f44d1ffb640 10 monclient: dump: 2024-03-29T10:40:41.594 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 1 2024-03-29T10:40:41.594 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:40:41.594 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:40:41.594 INFO:teuthology.orchestra.run.smithi196.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:40:41.594 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:41.594 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 1 2024-03-29T10:40:41.594 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:40:41.594 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:40:41.594 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:40:41.594 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-29T10:40:41.594 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.599+0000 7f44d9633640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:40:41.594 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.599+0000 7f44d1ffb640 1 -- 172.21.15.196:0/4069837445 <== mon.1 v1:172.21.15.196:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f44cc001bf0 con 0x7f44d40b4360 2024-03-29T10:40:41.595 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.599+0000 7f44d1ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:40:41.595 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.599+0000 7f44d9633640 10 monclient: get_monmap_and_config success 2024-03-29T10:40:41.595 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.599+0000 7f44d1ffb640 1 -- 172.21.15.196:0/4069837445 <== mon.1 v1:172.21.15.196:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f44cc002000 con 0x7f44d40b4360 2024-03-29T10:40:41.595 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.599+0000 7f44d1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:41.595 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.599+0000 7f44d9633640 4 set_mon_vals no callback set 2024-03-29T10:40:41.595 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.599+0000 7f44d1ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:40:41.595 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.599+0000 7f44d1ffb640 10 monclient: dump: 2024-03-29T10:40:41.595 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 1 2024-03-29T10:40:41.595 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:40:41.595 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:40:41.595 INFO:teuthology.orchestra.run.smithi196.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:40:41.595 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:41.595 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 1 2024-03-29T10:40:41.595 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:40:41.596 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:40:41.596 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:40:41.596 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-29T10:40:41.596 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.599+0000 7f44d9633640 10 monclient: shutdown 2024-03-29T10:40:41.596 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.599+0000 7f44d9633640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:40:41.596 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.599+0000 7f44d9633640 1 -- 172.21.15.196:0/4069837445 >> v1:172.21.15.196:6789/0 conn(0x7f44d40b4360 legacy=0x7f44d414d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:41.596 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.599+0000 7f44d9633640 1 -- 172.21.15.196:0/4069837445 shutdown_connections 2024-03-29T10:40:41.596 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.599+0000 7f44d9633640 1 -- 172.21.15.196:0/4069837445 >> 172.21.15.196:0/4069837445 conn(0x7f44d4066b00 msgr2=0x7f44d4066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:40:41.596 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.599+0000 7f44d9633640 1 -- 172.21.15.196:0/4069837445 shutdown_connections 2024-03-29T10:40:41.596 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.599+0000 7f44d9633640 1 -- 172.21.15.196:0/4069837445 wait complete. 2024-03-29T10:40:41.596 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.600+0000 7f44d9633640 10 monclient: build_initial_monmap 2024-03-29T10:40:41.596 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.600+0000 7f44d9633640 10 monclient: monmap: 2024-03-29T10:40:41.596 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 0 2024-03-29T10:40:41.596 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:40:41.597 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 0.000000 2024-03-29T10:40:41.597 INFO:teuthology.orchestra.run.smithi196.stderr:created 0.000000 2024-03-29T10:40:41.597 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 0 (unknown) 2024-03-29T10:40:41.597 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 1 2024-03-29T10:40:41.597 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-03-29T10:40:41.597 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-29T10:40:41.597 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.88:6790/0 mon.noname-c 2024-03-29T10:40:41.597 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-29T10:40:41.597 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.600+0000 7f44d9633640 1 librados: starting msgr at 2024-03-29T10:40:41.597 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.600+0000 7f44d9633640 1 librados: starting objecter 2024-03-29T10:40:41.597 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.600+0000 7f44d9633640 1 Processor -- start 2024-03-29T10:40:41.597 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.600+0000 7f44d9633640 1 -- start start 2024-03-29T10:40:41.597 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.600+0000 7f44d9633640 1 librados: setting wanted keys 2024-03-29T10:40:41.597 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.600+0000 7f44d9633640 1 librados: calling monclient init 2024-03-29T10:40:41.597 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.600+0000 7f44d9633640 10 monclient: init 2024-03-29T10:40:41.598 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.601+0000 7f44d9633640 10 monclient: _reopen_session rank -1 2024-03-29T10:40:41.598 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.601+0000 7f44d9633640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:40:41.598 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.601+0000 7f44d9633640 10 monclient(hunting): picked mon.noname-b con 0x7f44d409ac40 addr v1:172.21.15.196:6789/0 2024-03-29T10:40:41.598 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.601+0000 7f44d9633640 10 monclient(hunting): picked mon.noname-c con 0x7f44d41d7cd0 addr v1:172.21.15.88:6790/0 2024-03-29T10:40:41.598 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.601+0000 7f44d9633640 10 monclient(hunting): picked mon.noname-a con 0x7f44d40ad1e0 addr v1:172.21.15.88:6789/0 2024-03-29T10:40:41.598 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.601+0000 7f44d9633640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44d41d7130 con 0x7f44d40ad1e0 2024-03-29T10:40:41.598 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.601+0000 7f44d9633640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44d4139b10 con 0x7f44d409ac40 2024-03-29T10:40:41.598 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.601+0000 7f44d9633640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44d413ffa0 con 0x7f44d41d7cd0 2024-03-29T10:40:41.598 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.601+0000 7f44d9633640 10 monclient(hunting): _renew_subs 2024-03-29T10:40:41.598 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.601+0000 7f44d9633640 10 monclient(hunting): authenticate will time out at 787.916526s 2024-03-29T10:40:41.598 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.601+0000 7f44d2ffd640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f44d409ac40 0x7f44d41d75c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.196:39916/0 (socket says 172.21.15.196:39916) 2024-03-29T10:40:41.598 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.601+0000 7f44d2ffd640 1 -- 172.21.15.196:0/1811127491 learned_addr learned my addr 172.21.15.196:0/1811127491 (peer_addr_for_me v1:172.21.15.196:0/0) 2024-03-29T10:40:41.598 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.601+0000 7f44bb7fe640 10 client.?.objecter ms_handle_connect 0x7f44d409ac40 2024-03-29T10:40:41.598 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.601+0000 7f44bb7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:40:41.599 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.601+0000 7f44bb7fe640 10 client.?.objecter ms_handle_connect 0x7f44d41d7cd0 2024-03-29T10:40:41.599 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.601+0000 7f44bb7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:40:41.599 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.601+0000 7f44bb7fe640 1 -- 172.21.15.196:0/1811127491 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1488864413 0 0) 0x7f44d4139b10 con 0x7f44d409ac40 2024-03-29T10:40:41.599 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.601+0000 7f44bb7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:40:41.599 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.601+0000 7f44bb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:41.599 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.601+0000 7f44bb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:41.599 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.601+0000 7f44bb7fe640 10 monclient(hunting): my global_id is 4129 2024-03-29T10:40:41.599 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.601+0000 7f44bb7fe640 1 -- 172.21.15.196:0/1811127491 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f44c0004eb0 con 0x7f44d409ac40 2024-03-29T10:40:41.599 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.601+0000 7f44bb7fe640 10 client.?.objecter ms_handle_connect 0x7f44d40ad1e0 2024-03-29T10:40:41.599 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.601+0000 7f44bb7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:40:41.599 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.601+0000 7f44bb7fe640 1 -- 172.21.15.196:0/1811127491 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 416852931 0 0) 0x7f44d413ffa0 con 0x7f44d41d7cd0 2024-03-29T10:40:41.599 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.601+0000 7f44bb7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:40:41.599 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.601+0000 7f44bb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:41.599 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.601+0000 7f44bb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:41.600 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.601+0000 7f44bb7fe640 10 monclient(hunting): my global_id is 4127 2024-03-29T10:40:41.600 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.601+0000 7f44bb7fe640 1 -- 172.21.15.196:0/1811127491 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f44d4139b10 con 0x7f44d41d7cd0 2024-03-29T10:40:41.600 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.602+0000 7f44bb7fe640 1 -- 172.21.15.196:0/1811127491 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2935382260 0 0) 0x7f44d41d7130 con 0x7f44d40ad1e0 2024-03-29T10:40:41.600 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.602+0000 7f44bb7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:40:41.600 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.602+0000 7f44bb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:41.600 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.602+0000 7f44bb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:41.600 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.602+0000 7f44bb7fe640 10 monclient(hunting): my global_id is 4131 2024-03-29T10:40:41.600 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.602+0000 7f44bb7fe640 1 -- 172.21.15.196:0/1811127491 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f44d413ffa0 con 0x7f44d40ad1e0 2024-03-29T10:40:41.600 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.602+0000 7f44bb7fe640 1 -- 172.21.15.196:0/1811127491 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 293280104 0 0) 0x7f44c0004eb0 con 0x7f44d409ac40 2024-03-29T10:40:41.600 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.602+0000 7f44bb7fe640 1 -- 172.21.15.196:0/1811127491 --> v1:172.21.15.196:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f44d41d7130 con 0x7f44d409ac40 2024-03-29T10:40:41.600 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.602+0000 7f44bb7fe640 1 -- 172.21.15.196:0/1811127491 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f44c8001a10 con 0x7f44d409ac40 2024-03-29T10:40:41.600 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.602+0000 7f44bb7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:41.600 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.602+0000 7f44bb7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:40:41.600 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.602+0000 7f44bb7fe640 10 monclient(hunting): dump: 2024-03-29T10:40:41.600 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 1 2024-03-29T10:40:41.601 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:40:41.601 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:40:41.601 INFO:teuthology.orchestra.run.smithi196.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:40:41.601 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:41.601 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 1 2024-03-29T10:40:41.601 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:40:41.601 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:40:41.601 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:40:41.601 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-29T10:40:41.601 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.602+0000 7f44bb7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:40:41.601 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.602+0000 7f44bb7fe640 1 -- 172.21.15.196:0/1811127491 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2817347021 0 0) 0x7f44d4139b10 con 0x7f44d41d7cd0 2024-03-29T10:40:41.601 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.602+0000 7f44bb7fe640 1 -- 172.21.15.196:0/1811127491 --> v1:172.21.15.88:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f44c0004eb0 con 0x7f44d41d7cd0 2024-03-29T10:40:41.601 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.602+0000 7f44bb7fe640 1 -- 172.21.15.196:0/1811127491 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f44c4001920 con 0x7f44d41d7cd0 2024-03-29T10:40:41.601 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.602+0000 7f44bb7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:41.601 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.602+0000 7f44bb7fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:40:41.602 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.602+0000 7f44bb7fe640 10 monclient(hunting): dump: 2024-03-29T10:40:41.602 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 1 2024-03-29T10:40:41.602 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:40:41.602 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:40:41.602 INFO:teuthology.orchestra.run.smithi196.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:40:41.602 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:41.602 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 1 2024-03-29T10:40:41.602 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:40:41.602 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:40:41.602 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:40:41.602 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-29T10:40:41.602 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.602+0000 7f44bb7fe640 1 -- 172.21.15.196:0/1811127491 <== mon.1 v1:172.21.15.196:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1288541334 0 0) 0x7f44d41d7130 con 0x7f44d409ac40 2024-03-29T10:40:41.602 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.602+0000 7f44bb7fe640 1 -- 172.21.15.196:0/1811127491 >> v1:172.21.15.88:6790/0 conn(0x7f44d41d7cd0 legacy=0x7f44d4138cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:41.602 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.602+0000 7f44bb7fe640 1 -- 172.21.15.196:0/1811127491 >> v1:172.21.15.88:6789/0 conn(0x7f44d40ad1e0 legacy=0x7f44d4139400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:41.602 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.602+0000 7f44bb7fe640 10 monclient: _finish_hunting 0 2024-03-29T10:40:41.603 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.602+0000 7f44bb7fe640 1 monclient: found mon.b 2024-03-29T10:40:41.603 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.602+0000 7f44bb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:40:41.603 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.602+0000 7f44bb7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T10:40:41.603 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.602+0000 7f44bb7fe640 1 -- 172.21.15.196:0/1811127491 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44d40ae520 con 0x7f44d409ac40 2024-03-29T10:40:41.603 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.602+0000 7f44bb7fe640 10 monclient: _finish_auth 0 2024-03-29T10:40:41.603 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.602+0000 7f44bb7fe640 10 monclient: _check_auth_tickets 2024-03-29T10:40:41.603 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.602+0000 7f44bb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:40:41.603 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.602+0000 7f44d9633640 5 monclient: authenticate success, global_id 4129 2024-03-29T10:40:41.603 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.602+0000 7f44d9633640 10 monclient: _renew_subs 2024-03-29T10:40:41.603 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.602+0000 7f44d9633640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T10:40:41.603 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.602+0000 7f44d9633640 1 -- 172.21.15.196:0/1811127491 --> v1:172.21.15.196:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44d40ae800 con 0x7f44d409ac40 2024-03-29T10:40:41.603 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.602+0000 7f44d9633640 10 client.4129.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:40:41.603 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.602+0000 7f44d9633640 10 monclient: _renew_subs 2024-03-29T10:40:41.603 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.602+0000 7f44d9633640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T10:40:41.604 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.602+0000 7f44d9633640 1 -- 172.21.15.196:0/1811127491 --> v1:172.21.15.196:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f44d4142b70 con 0x7f44d409ac40 2024-03-29T10:40:41.604 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.602+0000 7f44d9633640 1 librados: init done 2024-03-29T10:40:41.604 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.602+0000 7f44bb7fe640 1 -- 172.21.15.196:0/1811127491 <== mon.1 v1:172.21.15.196:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f44c8001c70 con 0x7f44d409ac40 2024-03-29T10:40:41.604 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.602+0000 7f44bb7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:40:41.604 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.602+0000 7f44bb7fe640 1 -- 172.21.15.196:0/1811127491 <== mon.1 v1:172.21.15.196:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f44c8003a60 con 0x7f44d409ac40 2024-03-29T10:40:41.604 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.602+0000 7f44bb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:41.604 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.602+0000 7f44bb7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:40:41.604 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.602+0000 7f44d17fa640 4 set_mon_vals no callback set 2024-03-29T10:40:41.604 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.602+0000 7f44bb7fe640 10 monclient: dump: 2024-03-29T10:40:41.604 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 1 2024-03-29T10:40:41.604 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:40:41.604 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:40:41.604 INFO:teuthology.orchestra.run.smithi196.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:40:41.604 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:41.604 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 1 2024-03-29T10:40:41.604 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:40:41.605 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:40:41.605 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:40:41.605 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-29T10:40:41.605 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.603+0000 7f44bb7fe640 1 -- 172.21.15.196:0/1811127491 <== mon.1 v1:172.21.15.196:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f44c8003fd0 con 0x7f44d409ac40 2024-03-29T10:40:41.605 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.603+0000 7f44bb7fe640 1 -- 172.21.15.196:0/1811127491 <== mon.1 v1:172.21.15.196:6789/0 8 ==== osd_map(5..5 src has 1..5) v4 ==== 1070+0+0 (unknown 2687123227 0 0) 0x7f44c80067a0 con 0x7f44d409ac40 2024-03-29T10:40:41.605 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.603+0000 7f44bb7fe640 10 client.4129.objecter ms_dispatch 0x7f44d4000b90 osd_map(5..5 src has 1..5) v4 2024-03-29T10:40:41.605 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.603+0000 7f44bb7fe640 3 client.4129.objecter handle_osd_map got epochs [5,5] > 0 2024-03-29T10:40:41.605 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.603+0000 7f44bb7fe640 3 client.4129.objecter handle_osd_map decoding full epoch 5 2024-03-29T10:40:41.605 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.603+0000 7f44bb7fe640 20 client.4129.objecter dump_active .. 0 homeless 2024-03-29T10:40:41.605 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.603+0000 7f44d9633640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:40:41.605 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.603+0000 7f44d9633640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:40:41.605 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.603+0000 7f44d9633640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T10:40:41.605 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.603+0000 7f44d9633640 1 -- 172.21.15.196:0/1811127491 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44b4005080 con 0x7f44d409ac40 2024-03-29T10:40:41.605 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.605+0000 7f44bb7fe640 1 -- 172.21.15.196:0/1811127491 <== mon.1 v1:172.21.15.196:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f44c8004460 con 0x7f44d409ac40 2024-03-29T10:40:41.605 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.605+0000 7f44bb7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:40:41.606 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.605+0000 7f44bb7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:40:41.606 INFO:tasks.ceph.mgr.x.smithi196.stderr:2024-03-29T10:40:41.610+0000 7ff5dcc4b040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-29T10:40:41.606 INFO:tasks.ceph.mgr.y.smithi088.stderr:2024-03-29T10:40:41.608+0000 7fca56440040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-29T10:40:41.661 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.677+0000 7f44d9633640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "3ec4f045-c714-4a22-bd1a-e07330b45ab3", "id": 3}] 2024-03-29T10:40:41.661 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.677+0000 7f44d9633640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "3ec4f045-c714-4a22-bd1a-e07330b45ab3", "id": 3}] 2024-03-29T10:40:41.661 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.677+0000 7f44d9633640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T10:40:41.661 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.677+0000 7f44d9633640 1 -- 172.21.15.196:0/1811127491 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "osd new", "uuid": "3ec4f045-c714-4a22-bd1a-e07330b45ab3", "id": 3} v 0) v1 -- 0x7f44b40058b0 con 0x7f44d409ac40 2024-03-29T10:40:41.664 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.681+0000 7f44bb7fe640 1 -- 172.21.15.196:0/1811127491 <== mon.1 v1:172.21.15.196:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "3ec4f045-c714-4a22-bd1a-e07330b45ab3", "id": 3}]=0 v6) v1 ==== 112+0+2 (unknown 2812721415 0 525046306) 0x7f44c8006500 con 0x7f44d409ac40 2024-03-29T10:40:41.664 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.681+0000 7f44bb7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "3ec4f045-c714-4a22-bd1a-e07330b45ab3", "id": 3}] 2024-03-29T10:40:41.664 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.681+0000 7f44bb7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:40:41.665 INFO:teuthology.orchestra.run.smithi196.stdout:3 2024-03-29T10:40:41.666 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.682+0000 7f44d9633640 10 librados: watch_flush enter 2024-03-29T10:40:41.666 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.682+0000 7f44d9633640 10 librados: watch_flush exit 2024-03-29T10:40:41.666 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.682+0000 7f44d9633640 20 client.4129.objecter shutdown clearing up homeless session... 2024-03-29T10:40:41.666 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.682+0000 7f44d9633640 10 client.4129.objecter successfully canceled tick 2024-03-29T10:40:41.666 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.682+0000 7f44d9633640 10 monclient: shutdown 2024-03-29T10:40:41.666 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.682+0000 7f44d9633640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:40:41.666 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.682+0000 7f44d9633640 1 -- 172.21.15.196:0/1811127491 >> v1:172.21.15.196:6789/0 conn(0x7f44d409ac40 legacy=0x7f44d41d75c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:41.666 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.682+0000 7f44d9633640 1 -- 172.21.15.196:0/1811127491 shutdown_connections 2024-03-29T10:40:41.667 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.682+0000 7f44d9633640 1 -- 172.21.15.196:0/1811127491 >> 172.21.15.196:0/1811127491 conn(0x7f44d4066b00 msgr2=0x7f44d40b4a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:40:41.667 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.682+0000 7f44d9633640 1 -- 172.21.15.196:0/1811127491 shutdown_connections 2024-03-29T10:40:41.667 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.682+0000 7f44d9633640 1 -- 172.21.15.196:0/1811127491 wait complete. 2024-03-29T10:40:41.667 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.682+0000 7f44d9633640 1 librados: shutdown 2024-03-29T10:40:41.676 DEBUG:teuthology.orchestra.run.smithi196:> sudo ceph --cluster ceph osd new cddb9867-8e30-4951-9b13-3d1e78b9c45a 4 2024-03-29T10:40:41.707 INFO:tasks.ceph.mgr.y.smithi088.stderr:2024-03-29T10:40:41.724+0000 7fca56440040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-29T10:40:41.714 INFO:tasks.ceph.mgr.x.smithi196.stderr:2024-03-29T10:40:41.731+0000 7ff5dcc4b040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-29T10:40:41.777 INFO:tasks.ceph.mgr.y.smithi088.stderr:2024-03-29T10:40:41.793+0000 7fca56440040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-29T10:40:41.788 INFO:tasks.ceph.mgr.x.smithi196.stderr:2024-03-29T10:40:41.805+0000 7ff5dcc4b040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-29T10:40:41.814 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.830+0000 7f106aac1640 10 monclient: get_monmap_and_config 2024-03-29T10:40:41.814 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.830+0000 7f106aac1640 10 monclient: build_initial_monmap 2024-03-29T10:40:41.814 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.830+0000 7f106aac1640 10 monclient: monmap: 2024-03-29T10:40:41.814 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 0 2024-03-29T10:40:41.814 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:40:41.814 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 2024-03-29T10:40:41.831671+0000 2024-03-29T10:40:41.815 INFO:teuthology.orchestra.run.smithi196.stderr:created 2024-03-29T10:40:41.831671+0000 2024-03-29T10:40:41.815 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 0 (unknown) 2024-03-29T10:40:41.815 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 1 2024-03-29T10:40:41.815 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-03-29T10:40:41.815 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-29T10:40:41.815 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.88:6790/0 mon.noname-b 2024-03-29T10:40:41.815 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-29T10:40:41.823 INFO:tasks.ceph.mgr.y.smithi088.stderr:2024-03-29T10:40:41.839+0000 7fca56440040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-29T10:40:41.825 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.831+0000 7f106aac1640 1 Processor -- start 2024-03-29T10:40:41.825 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.831+0000 7f106aac1640 1 -- start start 2024-03-29T10:40:41.825 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.831+0000 7f106aac1640 10 monclient: init 2024-03-29T10:40:41.825 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.832+0000 7f106aac1640 10 monclient: _reopen_session rank -1 2024-03-29T10:40:41.825 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.832+0000 7f106aac1640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:40:41.825 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.832+0000 7f106aac1640 10 monclient(hunting): picked mon.noname-a con 0x7f106409ac40 addr v1:172.21.15.88:6789/0 2024-03-29T10:40:41.825 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.832+0000 7f106aac1640 10 monclient(hunting): picked mon.noname-c con 0x7f106409c870 addr v1:172.21.15.196:6789/0 2024-03-29T10:40:41.825 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.832+0000 7f106aac1640 10 monclient(hunting): picked mon.noname-b con 0x7f10640b4360 addr v1:172.21.15.88:6790/0 2024-03-29T10:40:41.825 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.832+0000 7f106aac1640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f106409dec0 con 0x7f106409ac40 2024-03-29T10:40:41.825 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.832+0000 7f106aac1640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f106409da30 con 0x7f106409c870 2024-03-29T10:40:41.825 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.832+0000 7f106aac1640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10640b39c0 con 0x7f10640b4360 2024-03-29T10:40:41.826 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.832+0000 7f106aac1640 10 monclient(hunting): _renew_subs 2024-03-29T10:40:41.826 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.832+0000 7f106aac1640 10 monclient(hunting): authenticate will time out at 788.147208s 2024-03-29T10:40:41.826 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.832+0000 7f1063fff640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f106409c870 0x7f106413f3a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.196:39932/0 (socket says 172.21.15.196:39932) 2024-03-29T10:40:41.826 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.832+0000 7f1063fff640 1 -- 172.21.15.196:0/2864410474 learned_addr learned my addr 172.21.15.196:0/2864410474 (peer_addr_for_me v1:172.21.15.196:0/0) 2024-03-29T10:40:41.826 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.832+0000 7f10637fe640 1 -- 172.21.15.196:0/2864410474 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2558160994 0 0) 0x7f106409da30 con 0x7f106409c870 2024-03-29T10:40:41.826 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.832+0000 7f10637fe640 10 monclient(hunting): _negotiate 2024-03-29T10:40:41.826 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.832+0000 7f10637fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:41.826 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.832+0000 7f10637fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:41.826 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.832+0000 7f10637fe640 10 monclient(hunting): my global_id is 4132 2024-03-29T10:40:41.826 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.832+0000 7f10637fe640 1 -- 172.21.15.196:0/2864410474 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1054003520 con 0x7f106409c870 2024-03-29T10:40:41.826 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.832+0000 7f10637fe640 1 -- 172.21.15.196:0/2864410474 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2428309616 0 0) 0x7f1058001110 con 0x7f106409ac40 2024-03-29T10:40:41.826 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.832+0000 7f10637fe640 10 monclient(hunting): _negotiate 2024-03-29T10:40:41.826 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.832+0000 7f10637fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:41.826 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.832+0000 7f10637fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:41.826 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.832+0000 7f10637fe640 10 monclient(hunting): my global_id is 4134 2024-03-29T10:40:41.827 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.832+0000 7f10637fe640 1 -- 172.21.15.196:0/2864410474 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f106409da30 con 0x7f106409ac40 2024-03-29T10:40:41.827 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.832+0000 7f10637fe640 1 -- 172.21.15.196:0/2864410474 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3750971686 0 0) 0x7f10640b39c0 con 0x7f10640b4360 2024-03-29T10:40:41.827 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.832+0000 7f10637fe640 10 monclient(hunting): _negotiate 2024-03-29T10:40:41.827 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.832+0000 7f10637fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:41.827 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.832+0000 7f10637fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:41.827 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.832+0000 7f10637fe640 10 monclient(hunting): my global_id is 4130 2024-03-29T10:40:41.827 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.832+0000 7f10637fe640 1 -- 172.21.15.196:0/2864410474 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1058001110 con 0x7f10640b4360 2024-03-29T10:40:41.827 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.832+0000 7f10637fe640 1 -- 172.21.15.196:0/2864410474 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3638274924 0 0) 0x7f1054003520 con 0x7f106409c870 2024-03-29T10:40:41.827 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.832+0000 7f10637fe640 1 -- 172.21.15.196:0/2864410474 >> v1:172.21.15.88:6790/0 conn(0x7f10640b4360 legacy=0x7f106414d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:41.827 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.832+0000 7f10637fe640 1 -- 172.21.15.196:0/2864410474 >> v1:172.21.15.88:6789/0 conn(0x7f106409ac40 legacy=0x7f10640b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:41.827 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.832+0000 7f10637fe640 10 monclient: _finish_hunting 0 2024-03-29T10:40:41.827 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.832+0000 7f10637fe640 1 monclient: found mon.noname-c 2024-03-29T10:40:41.827 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.832+0000 7f10637fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:40:41.827 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.832+0000 7f10637fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.196:6789/0 2024-03-29T10:40:41.828 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.832+0000 7f10637fe640 1 -- 172.21.15.196:0/2864410474 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10640b5160 con 0x7f106409c870 2024-03-29T10:40:41.828 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.832+0000 7f10637fe640 10 monclient: _finish_auth 0 2024-03-29T10:40:41.828 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.832+0000 7f10637fe640 10 monclient: _check_auth_tickets 2024-03-29T10:40:41.828 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.832+0000 7f10637fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:40:41.828 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.832+0000 7f10637fe640 1 -- 172.21.15.196:0/2864410474 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f1050001690 con 0x7f106409c870 2024-03-29T10:40:41.828 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.832+0000 7f106aac1640 5 monclient: authenticate success, global_id 4132 2024-03-29T10:40:41.828 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.832+0000 7f106aac1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:40:41.828 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.832+0000 7f10637fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:41.828 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.832+0000 7f10637fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:40:41.828 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.832+0000 7f10637fe640 10 monclient: dump: 2024-03-29T10:40:41.828 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 1 2024-03-29T10:40:41.828 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:40:41.828 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:40:41.828 INFO:teuthology.orchestra.run.smithi196.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:40:41.828 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:41.828 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 1 2024-03-29T10:40:41.829 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:40:41.829 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:40:41.829 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:40:41.829 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-29T10:40:41.829 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.832+0000 7f106aac1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:40:41.829 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.832+0000 7f10637fe640 1 -- 172.21.15.196:0/2864410474 <== mon.1 v1:172.21.15.196:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1050001870 con 0x7f106409c870 2024-03-29T10:40:41.829 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.832+0000 7f10637fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:40:41.829 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.832+0000 7f10637fe640 1 -- 172.21.15.196:0/2864410474 <== mon.1 v1:172.21.15.196:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f1050001c80 con 0x7f106409c870 2024-03-29T10:40:41.829 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.832+0000 7f106aac1640 10 monclient: get_monmap_and_config success 2024-03-29T10:40:41.829 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.832+0000 7f106aac1640 4 set_mon_vals no callback set 2024-03-29T10:40:41.829 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.832+0000 7f10637fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:41.829 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.832+0000 7f10637fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:40:41.829 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.832+0000 7f10637fe640 10 monclient: dump: 2024-03-29T10:40:41.829 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 1 2024-03-29T10:40:41.829 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:40:41.830 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:40:41.830 INFO:teuthology.orchestra.run.smithi196.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:40:41.830 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:41.830 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 1 2024-03-29T10:40:41.830 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:40:41.830 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:40:41.830 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:40:41.830 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-29T10:40:41.830 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.833+0000 7f106aac1640 10 monclient: shutdown 2024-03-29T10:40:41.830 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.833+0000 7f106aac1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:40:41.830 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.833+0000 7f106aac1640 1 -- 172.21.15.196:0/2864410474 >> v1:172.21.15.196:6789/0 conn(0x7f106409c870 legacy=0x7f106413f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:41.830 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.833+0000 7f106aac1640 1 -- 172.21.15.196:0/2864410474 shutdown_connections 2024-03-29T10:40:41.830 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.833+0000 7f106aac1640 1 -- 172.21.15.196:0/2864410474 >> 172.21.15.196:0/2864410474 conn(0x7f1064066b00 msgr2=0x7f1064066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:40:41.831 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.833+0000 7f106aac1640 1 -- 172.21.15.196:0/2864410474 shutdown_connections 2024-03-29T10:40:41.831 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.833+0000 7f106aac1640 1 -- 172.21.15.196:0/2864410474 wait complete. 2024-03-29T10:40:41.831 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.834+0000 7f106aac1640 10 monclient: build_initial_monmap 2024-03-29T10:40:41.831 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.834+0000 7f106aac1640 10 monclient: monmap: 2024-03-29T10:40:41.831 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 0 2024-03-29T10:40:41.831 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:40:41.831 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 0.000000 2024-03-29T10:40:41.831 INFO:teuthology.orchestra.run.smithi196.stderr:created 0.000000 2024-03-29T10:40:41.831 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 0 (unknown) 2024-03-29T10:40:41.831 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 1 2024-03-29T10:40:41.831 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-03-29T10:40:41.831 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-29T10:40:41.831 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.88:6790/0 mon.noname-c 2024-03-29T10:40:41.831 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-29T10:40:41.831 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.834+0000 7f106aac1640 1 librados: starting msgr at 2024-03-29T10:40:41.832 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.834+0000 7f106aac1640 1 librados: starting objecter 2024-03-29T10:40:41.832 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.834+0000 7f106aac1640 1 Processor -- start 2024-03-29T10:40:41.832 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.834+0000 7f106aac1640 1 -- start start 2024-03-29T10:40:41.832 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.834+0000 7f106aac1640 1 librados: setting wanted keys 2024-03-29T10:40:41.832 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.834+0000 7f106aac1640 1 librados: calling monclient init 2024-03-29T10:40:41.832 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.834+0000 7f106aac1640 10 monclient: init 2024-03-29T10:40:41.832 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.835+0000 7f106aac1640 10 monclient: _reopen_session rank -1 2024-03-29T10:40:41.832 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.835+0000 7f106aac1640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:40:41.832 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.835+0000 7f106aac1640 10 monclient(hunting): picked mon.noname-c con 0x7f106409ac40 addr v1:172.21.15.88:6790/0 2024-03-29T10:40:41.832 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.835+0000 7f106aac1640 10 monclient(hunting): picked mon.noname-b con 0x7f10641d7bc0 addr v1:172.21.15.196:6789/0 2024-03-29T10:40:41.832 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.835+0000 7f106aac1640 10 monclient(hunting): picked mon.noname-a con 0x7f10640b17f0 addr v1:172.21.15.88:6789/0 2024-03-29T10:40:41.832 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.835+0000 7f106aac1640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f106413b890 con 0x7f10640b17f0 2024-03-29T10:40:41.832 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.835+0000 7f106aac1640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10641d76a0 con 0x7f10641d7bc0 2024-03-29T10:40:41.833 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.835+0000 7f106aac1640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10641406b0 con 0x7f106409ac40 2024-03-29T10:40:41.833 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.835+0000 7f106aac1640 10 monclient(hunting): _renew_subs 2024-03-29T10:40:41.833 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.835+0000 7f106aac1640 10 monclient(hunting): authenticate will time out at 788.150174s 2024-03-29T10:40:41.833 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.835+0000 7f1063fff640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f10641d7bc0 0x7f106413aa70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.196:39936/0 (socket says 172.21.15.196:39936) 2024-03-29T10:40:41.833 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.835+0000 7f1063fff640 1 -- 172.21.15.196:0/2549111541 learned_addr learned my addr 172.21.15.196:0/2549111541 (peer_addr_for_me v1:172.21.15.196:0/0) 2024-03-29T10:40:41.833 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.835+0000 7f10617fa640 10 client.?.objecter ms_handle_connect 0x7f10641d7bc0 2024-03-29T10:40:41.833 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.835+0000 7f10617fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:40:41.833 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.835+0000 7f10617fa640 10 client.?.objecter ms_handle_connect 0x7f10640b17f0 2024-03-29T10:40:41.833 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.835+0000 7f10617fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:40:41.833 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.835+0000 7f10617fa640 10 client.?.objecter ms_handle_connect 0x7f106409ac40 2024-03-29T10:40:41.833 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.835+0000 7f10617fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:40:41.833 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.835+0000 7f10617fa640 1 -- 172.21.15.196:0/2549111541 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1095089550 0 0) 0x7f10641d76a0 con 0x7f10641d7bc0 2024-03-29T10:40:41.833 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.835+0000 7f10617fa640 10 monclient(hunting): _negotiate 2024-03-29T10:40:41.833 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.835+0000 7f10617fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:41.833 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.835+0000 7f10617fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:41.834 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.835+0000 7f10617fa640 10 monclient(hunting): my global_id is 4135 2024-03-29T10:40:41.834 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.835+0000 7f10617fa640 1 -- 172.21.15.196:0/2549111541 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1048004eb0 con 0x7f10641d7bc0 2024-03-29T10:40:41.834 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.835+0000 7f10617fa640 1 -- 172.21.15.196:0/2549111541 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 538186818 0 0) 0x7f106413b890 con 0x7f10640b17f0 2024-03-29T10:40:41.834 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.835+0000 7f10617fa640 10 monclient(hunting): _negotiate 2024-03-29T10:40:41.834 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.835+0000 7f10617fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:41.834 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.835+0000 7f10617fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:41.834 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.835+0000 7f10617fa640 10 monclient(hunting): my global_id is 4137 2024-03-29T10:40:41.834 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.835+0000 7f10617fa640 1 -- 172.21.15.196:0/2549111541 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f10641d76a0 con 0x7f10640b17f0 2024-03-29T10:40:41.834 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.835+0000 7f10617fa640 1 -- 172.21.15.196:0/2549111541 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1700049448 0 0) 0x7f10641406b0 con 0x7f106409ac40 2024-03-29T10:40:41.834 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.835+0000 7f10617fa640 10 monclient(hunting): _negotiate 2024-03-29T10:40:41.834 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.835+0000 7f10617fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:41.834 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.835+0000 7f10617fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:41.834 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.835+0000 7f10617fa640 10 monclient(hunting): my global_id is 4133 2024-03-29T10:40:41.834 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.835+0000 7f10617fa640 1 -- 172.21.15.196:0/2549111541 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f106413b890 con 0x7f106409ac40 2024-03-29T10:40:41.835 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.835+0000 7f10617fa640 1 -- 172.21.15.196:0/2549111541 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3014587780 0 0) 0x7f10641d76a0 con 0x7f10640b17f0 2024-03-29T10:40:41.835 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.835+0000 7f10617fa640 1 -- 172.21.15.196:0/2549111541 --> v1:172.21.15.88:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f10641406b0 con 0x7f10640b17f0 2024-03-29T10:40:41.835 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.835+0000 7f10617fa640 1 -- 172.21.15.196:0/2549111541 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f105c001a80 con 0x7f10640b17f0 2024-03-29T10:40:41.835 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.835+0000 7f10617fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:41.835 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.835+0000 7f10617fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:40:41.835 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.835+0000 7f10617fa640 10 monclient(hunting): dump: 2024-03-29T10:40:41.835 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 1 2024-03-29T10:40:41.835 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:40:41.835 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:40:41.835 INFO:teuthology.orchestra.run.smithi196.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:40:41.835 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:41.835 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 1 2024-03-29T10:40:41.835 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:40:41.835 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:40:41.835 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:40:41.836 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-29T10:40:41.836 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.835+0000 7f10617fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:40:41.836 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.835+0000 7f10617fa640 1 -- 172.21.15.196:0/2549111541 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4119110540 0 0) 0x7f1048004eb0 con 0x7f10641d7bc0 2024-03-29T10:40:41.836 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.835+0000 7f10617fa640 1 -- 172.21.15.196:0/2549111541 --> v1:172.21.15.196:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f10641d76a0 con 0x7f10641d7bc0 2024-03-29T10:40:41.836 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.835+0000 7f10617fa640 1 -- 172.21.15.196:0/2549111541 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f1050001b80 con 0x7f10641d7bc0 2024-03-29T10:40:41.836 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.835+0000 7f10617fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:41.836 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.835+0000 7f10617fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T10:40:41.836 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.835+0000 7f10617fa640 10 monclient(hunting): dump: 2024-03-29T10:40:41.836 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 1 2024-03-29T10:40:41.836 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:40:41.836 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:40:41.836 INFO:teuthology.orchestra.run.smithi196.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:40:41.836 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:41.836 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 1 2024-03-29T10:40:41.837 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:40:41.837 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:40:41.837 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:40:41.837 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-29T10:40:41.837 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.835+0000 7f10617fa640 1 -- 172.21.15.196:0/2549111541 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 260208305 0 0) 0x7f106413b890 con 0x7f106409ac40 2024-03-29T10:40:41.837 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.835+0000 7f10617fa640 1 -- 172.21.15.196:0/2549111541 --> v1:172.21.15.88:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1048004eb0 con 0x7f106409ac40 2024-03-29T10:40:41.837 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.835+0000 7f10617fa640 1 -- 172.21.15.196:0/2549111541 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f10580018a0 con 0x7f106409ac40 2024-03-29T10:40:41.837 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.835+0000 7f10617fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:41.837 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.835+0000 7f10617fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:40:41.837 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.835+0000 7f10617fa640 10 monclient(hunting): dump: 2024-03-29T10:40:41.837 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 1 2024-03-29T10:40:41.837 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:40:41.837 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:40:41.837 INFO:teuthology.orchestra.run.smithi196.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:40:41.838 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:41.838 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 1 2024-03-29T10:40:41.838 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:40:41.838 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:40:41.838 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:40:41.838 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-29T10:40:41.838 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.835+0000 7f10617fa640 1 -- 172.21.15.196:0/2549111541 <== mon.0 v1:172.21.15.88:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1026084398 0 0) 0x7f10641406b0 con 0x7f10640b17f0 2024-03-29T10:40:41.838 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.835+0000 7f10617fa640 1 -- 172.21.15.196:0/2549111541 >> v1:172.21.15.88:6790/0 conn(0x7f106409ac40 legacy=0x7f106413ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:41.838 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.835+0000 7f10617fa640 1 -- 172.21.15.196:0/2549111541 >> v1:172.21.15.196:6789/0 conn(0x7f10641d7bc0 legacy=0x7f106413aa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:41.838 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.835+0000 7f10617fa640 10 monclient: _finish_hunting 0 2024-03-29T10:40:41.838 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.835+0000 7f10617fa640 1 monclient: found mon.a 2024-03-29T10:40:41.838 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.835+0000 7f10617fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:40:41.838 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.835+0000 7f10617fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-03-29T10:40:41.838 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.835+0000 7f10617fa640 1 -- 172.21.15.196:0/2549111541 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1064140840 con 0x7f10640b17f0 2024-03-29T10:40:41.839 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.835+0000 7f10617fa640 10 monclient: _finish_auth 0 2024-03-29T10:40:41.839 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.835+0000 7f10617fa640 10 monclient: _check_auth_tickets 2024-03-29T10:40:41.839 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.835+0000 7f10617fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:40:41.839 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.835+0000 7f106aac1640 5 monclient: authenticate success, global_id 4137 2024-03-29T10:40:41.839 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.835+0000 7f106aac1640 10 monclient: _renew_subs 2024-03-29T10:40:41.839 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.835+0000 7f106aac1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-03-29T10:40:41.839 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.835+0000 7f106aac1640 1 -- 172.21.15.196:0/2549111541 --> v1:172.21.15.88:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10640b6f40 con 0x7f10640b17f0 2024-03-29T10:40:41.839 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.836+0000 7f106aac1640 10 client.4137.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:40:41.839 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.836+0000 7f106aac1640 10 monclient: _renew_subs 2024-03-29T10:40:41.839 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.836+0000 7f106aac1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-03-29T10:40:41.839 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.836+0000 7f106aac1640 1 -- 172.21.15.196:0/2549111541 --> v1:172.21.15.88:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f10641d6f40 con 0x7f10640b17f0 2024-03-29T10:40:41.839 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.836+0000 7f106aac1640 1 librados: init done 2024-03-29T10:40:41.839 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.836+0000 7f10617fa640 1 -- 172.21.15.196:0/2549111541 <== mon.0 v1:172.21.15.88:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f105c002010 con 0x7f10640b17f0 2024-03-29T10:40:41.840 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.836+0000 7f10617fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:40:41.840 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.836+0000 7f10617fa640 1 -- 172.21.15.196:0/2549111541 <== mon.0 v1:172.21.15.88:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f105c003b30 con 0x7f10640b17f0 2024-03-29T10:40:41.840 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.836+0000 7f10617fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:41.840 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.836+0000 7f10617fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:40:41.840 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.836+0000 7f1062ffd640 4 set_mon_vals no callback set 2024-03-29T10:40:41.840 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.836+0000 7f10617fa640 10 monclient: dump: 2024-03-29T10:40:41.840 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 1 2024-03-29T10:40:41.840 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:40:41.840 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:40:41.840 INFO:teuthology.orchestra.run.smithi196.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:40:41.840 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:41.840 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 1 2024-03-29T10:40:41.840 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:40:41.840 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:40:41.841 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:40:41.841 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-29T10:40:41.841 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.836+0000 7f10617fa640 1 -- 172.21.15.196:0/2549111541 <== mon.0 v1:172.21.15.88:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f105c004020 con 0x7f10640b17f0 2024-03-29T10:40:41.841 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.836+0000 7f10617fa640 1 -- 172.21.15.196:0/2549111541 <== mon.0 v1:172.21.15.88:6789/0 8 ==== osd_map(6..6 src has 1..6) v4 ==== 1185+0+0 (unknown 931833034 0 0) 0x7f105c0047e0 con 0x7f10640b17f0 2024-03-29T10:40:41.841 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.836+0000 7f10617fa640 10 client.4137.objecter ms_dispatch 0x7f1064000b90 osd_map(6..6 src has 1..6) v4 2024-03-29T10:40:41.841 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.836+0000 7f10617fa640 3 client.4137.objecter handle_osd_map got epochs [6,6] > 0 2024-03-29T10:40:41.841 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.836+0000 7f10617fa640 3 client.4137.objecter handle_osd_map decoding full epoch 6 2024-03-29T10:40:41.841 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.836+0000 7f10617fa640 20 client.4137.objecter dump_active .. 0 homeless 2024-03-29T10:40:41.841 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.836+0000 7f106aac1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:40:41.841 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.836+0000 7f106aac1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:40:41.841 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.836+0000 7f106aac1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-03-29T10:40:41.841 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.836+0000 7f106aac1640 1 -- 172.21.15.196:0/2549111541 --> v1:172.21.15.88:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1044005080 con 0x7f10640b17f0 2024-03-29T10:40:41.841 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.839+0000 7f10617fa640 1 -- 172.21.15.196:0/2549111541 <== mon.0 v1:172.21.15.88:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f105c004b70 con 0x7f10640b17f0 2024-03-29T10:40:41.841 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.839+0000 7f10617fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:40:41.841 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.839+0000 7f10617fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:40:41.842 INFO:tasks.ceph.mgr.x.smithi196.stderr:2024-03-29T10:40:41.854+0000 7ff5dcc4b040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-29T10:40:41.875 INFO:tasks.ceph.mgr.y.smithi088.stderr:2024-03-29T10:40:41.891+0000 7fca56440040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-29T10:40:41.893 INFO:tasks.ceph.mgr.x.smithi196.stderr:2024-03-29T10:40:41.908+0000 7ff5dcc4b040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-29T10:40:41.895 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.911+0000 7f106aac1640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "cddb9867-8e30-4951-9b13-3d1e78b9c45a", "id": 4}] 2024-03-29T10:40:41.895 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.911+0000 7f106aac1640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "cddb9867-8e30-4951-9b13-3d1e78b9c45a", "id": 4}] 2024-03-29T10:40:41.895 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.911+0000 7f106aac1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-03-29T10:40:41.895 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.911+0000 7f106aac1640 1 -- 172.21.15.196:0/2549111541 --> v1:172.21.15.88:6789/0 -- mon_command({"prefix": "osd new", "uuid": "cddb9867-8e30-4951-9b13-3d1e78b9c45a", "id": 4} v 0) v1 -- 0x7f10440058b0 con 0x7f10640b17f0 2024-03-29T10:40:41.896 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.912+0000 7f10617fa640 1 -- 172.21.15.196:0/2549111541 <== mon.0 v1:172.21.15.88:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "cddb9867-8e30-4951-9b13-3d1e78b9c45a", "id": 4}]=0 v7) v1 ==== 112+0+2 (unknown 3881158969 0 1697012583) 0x7f105c0042d0 con 0x7f10640b17f0 2024-03-29T10:40:41.896 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.912+0000 7f10617fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "cddb9867-8e30-4951-9b13-3d1e78b9c45a", "id": 4}] 2024-03-29T10:40:41.896 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.912+0000 7f10617fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:40:41.896 INFO:teuthology.orchestra.run.smithi196.stdout:4 2024-03-29T10:40:41.898 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.913+0000 7f106aac1640 10 librados: watch_flush enter 2024-03-29T10:40:41.898 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.913+0000 7f106aac1640 10 librados: watch_flush exit 2024-03-29T10:40:41.898 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.913+0000 7f106aac1640 20 client.4137.objecter shutdown clearing up homeless session... 2024-03-29T10:40:41.898 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.913+0000 7f106aac1640 10 client.4137.objecter successfully canceled tick 2024-03-29T10:40:41.898 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.913+0000 7f106aac1640 10 monclient: shutdown 2024-03-29T10:40:41.898 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.913+0000 7f106aac1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:40:41.898 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.913+0000 7f106aac1640 1 -- 172.21.15.196:0/2549111541 >> v1:172.21.15.88:6789/0 conn(0x7f10640b17f0 legacy=0x7f106413b180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:41.898 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.913+0000 7f106aac1640 1 -- 172.21.15.196:0/2549111541 shutdown_connections 2024-03-29T10:40:41.898 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.913+0000 7f106aac1640 1 -- 172.21.15.196:0/2549111541 >> 172.21.15.196:0/2549111541 conn(0x7f1064066b00 msgr2=0x7f106413d800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:40:41.898 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.913+0000 7f106aac1640 1 -- 172.21.15.196:0/2549111541 shutdown_connections 2024-03-29T10:40:41.898 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.914+0000 7f106aac1640 1 -- 172.21.15.196:0/2549111541 wait complete. 2024-03-29T10:40:41.898 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:41.914+0000 7f106aac1640 1 librados: shutdown 2024-03-29T10:40:41.907 DEBUG:teuthology.orchestra.run.smithi196:> sudo ceph --cluster ceph osd new dc6d9911-785d-432f-8b53-56a1ccab302f 5 2024-03-29T10:40:41.922 INFO:tasks.ceph.mgr.y.smithi088.stderr:2024-03-29T10:40:41.939+0000 7fca56440040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-29T10:40:41.943 INFO:tasks.ceph.mgr.x.smithi196.stderr:2024-03-29T10:40:41.959+0000 7ff5dcc4b040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-29T10:40:42.007 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.023+0000 7f6e3254d640 10 monclient: get_monmap_and_config 2024-03-29T10:40:42.007 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.023+0000 7f6e3254d640 10 monclient: build_initial_monmap 2024-03-29T10:40:42.007 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.023+0000 7f6e3254d640 10 monclient: monmap: 2024-03-29T10:40:42.008 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 0 2024-03-29T10:40:42.008 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:40:42.008 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 2024-03-29T10:40:42.024766+0000 2024-03-29T10:40:42.008 INFO:teuthology.orchestra.run.smithi196.stderr:created 2024-03-29T10:40:42.024766+0000 2024-03-29T10:40:42.008 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 0 (unknown) 2024-03-29T10:40:42.008 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 1 2024-03-29T10:40:42.008 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-03-29T10:40:42.008 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-29T10:40:42.008 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.88:6790/0 mon.noname-b 2024-03-29T10:40:42.008 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-29T10:40:42.019 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.024+0000 7f6e3254d640 1 Processor -- start 2024-03-29T10:40:42.019 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.024+0000 7f6e3254d640 1 -- start start 2024-03-29T10:40:42.019 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.024+0000 7f6e3254d640 10 monclient: init 2024-03-29T10:40:42.019 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.025+0000 7f6e3254d640 10 monclient: _reopen_session rank -1 2024-03-29T10:40:42.019 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.025+0000 7f6e3254d640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:40:42.019 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.025+0000 7f6e3254d640 10 monclient(hunting): picked mon.noname-a con 0x7f6e2c09ac40 addr v1:172.21.15.88:6789/0 2024-03-29T10:40:42.020 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.025+0000 7f6e3254d640 10 monclient(hunting): picked mon.noname-b con 0x7f6e2c09c870 addr v1:172.21.15.88:6790/0 2024-03-29T10:40:42.020 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.025+0000 7f6e3254d640 10 monclient(hunting): picked mon.noname-c con 0x7f6e2c0b4360 addr v1:172.21.15.196:6789/0 2024-03-29T10:40:42.020 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.025+0000 7f6e3254d640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e2c09dec0 con 0x7f6e2c09ac40 2024-03-29T10:40:42.020 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.025+0000 7f6e3254d640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e2c09da30 con 0x7f6e2c0b4360 2024-03-29T10:40:42.020 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.025+0000 7f6e3254d640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e2c0b39c0 con 0x7f6e2c09c870 2024-03-29T10:40:42.020 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.025+0000 7f6e3254d640 10 monclient(hunting): _renew_subs 2024-03-29T10:40:42.020 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.025+0000 7f6e3254d640 10 monclient(hunting): authenticate will time out at 788.340337s 2024-03-29T10:40:42.020 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.025+0000 7f6e30aea640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f6e2c0b4360 0x7f6e2c14d9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.196:39948/0 (socket says 172.21.15.196:39948) 2024-03-29T10:40:42.020 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.025+0000 7f6e30aea640 1 -- 172.21.15.196:0/4183442524 learned_addr learned my addr 172.21.15.196:0/4183442524 (peer_addr_for_me v1:172.21.15.196:0/0) 2024-03-29T10:40:42.020 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.025+0000 7f6e2affd640 1 -- 172.21.15.196:0/4183442524 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1565458556 0 0) 0x7f6e2c09da30 con 0x7f6e2c0b4360 2024-03-29T10:40:42.020 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.025+0000 7f6e2affd640 10 monclient(hunting): _negotiate 2024-03-29T10:40:42.020 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.025+0000 7f6e2affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:42.021 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.025+0000 7f6e2affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:42.021 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.025+0000 7f6e2affd640 10 monclient(hunting): my global_id is 4138 2024-03-29T10:40:42.021 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.025+0000 7f6e2affd640 1 -- 172.21.15.196:0/4183442524 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e14003520 con 0x7f6e2c0b4360 2024-03-29T10:40:42.021 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.025+0000 7f6e2affd640 1 -- 172.21.15.196:0/4183442524 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 40689013 0 0) 0x7f6e20001040 con 0x7f6e2c09ac40 2024-03-29T10:40:42.021 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.025+0000 7f6e2affd640 10 monclient(hunting): _negotiate 2024-03-29T10:40:42.021 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.025+0000 7f6e2affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:42.021 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.025+0000 7f6e2affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:42.021 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.025+0000 7f6e2affd640 10 monclient(hunting): my global_id is 4140 2024-03-29T10:40:42.021 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.025+0000 7f6e2affd640 1 -- 172.21.15.196:0/4183442524 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e2c09da30 con 0x7f6e2c09ac40 2024-03-29T10:40:42.021 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.025+0000 7f6e2affd640 1 -- 172.21.15.196:0/4183442524 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1492652947 0 0) 0x7f6e2c0b39c0 con 0x7f6e2c09c870 2024-03-29T10:40:42.021 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.025+0000 7f6e2affd640 10 monclient(hunting): _negotiate 2024-03-29T10:40:42.021 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.025+0000 7f6e2affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:42.022 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.025+0000 7f6e2affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:42.022 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.025+0000 7f6e2affd640 10 monclient(hunting): my global_id is 4136 2024-03-29T10:40:42.022 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.025+0000 7f6e2affd640 1 -- 172.21.15.196:0/4183442524 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e20001040 con 0x7f6e2c09c870 2024-03-29T10:40:42.022 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.025+0000 7f6e2affd640 1 -- 172.21.15.196:0/4183442524 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3885637919 0 0) 0x7f6e14003520 con 0x7f6e2c0b4360 2024-03-29T10:40:42.022 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.025+0000 7f6e2affd640 1 -- 172.21.15.196:0/4183442524 >> v1:172.21.15.88:6790/0 conn(0x7f6e2c09c870 legacy=0x7f6e2c13f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:42.022 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.025+0000 7f6e2affd640 1 -- 172.21.15.196:0/4183442524 >> v1:172.21.15.88:6789/0 conn(0x7f6e2c09ac40 legacy=0x7f6e2c0b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:42.022 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.025+0000 7f6e2affd640 10 monclient: _finish_hunting 0 2024-03-29T10:40:42.022 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.025+0000 7f6e2affd640 1 monclient: found mon.noname-c 2024-03-29T10:40:42.022 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.025+0000 7f6e2affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:40:42.022 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.025+0000 7f6e2affd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.196:6789/0 2024-03-29T10:40:42.022 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.025+0000 7f6e2affd640 1 -- 172.21.15.196:0/4183442524 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e2c0b5160 con 0x7f6e2c0b4360 2024-03-29T10:40:42.022 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.025+0000 7f6e2affd640 10 monclient: _finish_auth 0 2024-03-29T10:40:42.023 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.025+0000 7f6e2affd640 10 monclient: _check_auth_tickets 2024-03-29T10:40:42.023 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.025+0000 7f6e2affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:40:42.023 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.025+0000 7f6e3254d640 5 monclient: authenticate success, global_id 4138 2024-03-29T10:40:42.023 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.025+0000 7f6e2affd640 1 -- 172.21.15.196:0/4183442524 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f6e24001a10 con 0x7f6e2c0b4360 2024-03-29T10:40:42.023 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.025+0000 7f6e3254d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:40:42.023 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.025+0000 7f6e2affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:42.023 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.026+0000 7f6e2affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:40:42.023 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.026+0000 7f6e2affd640 10 monclient: dump: 2024-03-29T10:40:42.023 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 1 2024-03-29T10:40:42.023 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:40:42.023 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:40:42.023 INFO:teuthology.orchestra.run.smithi196.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:40:42.024 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:42.024 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 1 2024-03-29T10:40:42.024 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:40:42.024 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:40:42.024 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:40:42.024 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-29T10:40:42.024 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.026+0000 7f6e3254d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:40:42.024 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.026+0000 7f6e2affd640 1 -- 172.21.15.196:0/4183442524 <== mon.1 v1:172.21.15.196:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6e24001bf0 con 0x7f6e2c0b4360 2024-03-29T10:40:42.024 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.026+0000 7f6e2affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:40:42.024 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.026+0000 7f6e2affd640 1 -- 172.21.15.196:0/4183442524 <== mon.1 v1:172.21.15.196:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f6e24002000 con 0x7f6e2c0b4360 2024-03-29T10:40:42.024 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.026+0000 7f6e3254d640 10 monclient: get_monmap_and_config success 2024-03-29T10:40:42.024 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.026+0000 7f6e3254d640 4 set_mon_vals no callback set 2024-03-29T10:40:42.024 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.026+0000 7f6e2affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:42.025 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.026+0000 7f6e2affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:40:42.025 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.026+0000 7f6e2affd640 10 monclient: dump: 2024-03-29T10:40:42.025 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 1 2024-03-29T10:40:42.025 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:40:42.025 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:40:42.025 INFO:teuthology.orchestra.run.smithi196.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:40:42.025 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:42.025 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 1 2024-03-29T10:40:42.025 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:40:42.025 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:40:42.026 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:40:42.026 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-29T10:40:42.026 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.026+0000 7f6e3254d640 10 monclient: shutdown 2024-03-29T10:40:42.026 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.026+0000 7f6e3254d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:40:42.026 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.026+0000 7f6e3254d640 1 -- 172.21.15.196:0/4183442524 >> v1:172.21.15.196:6789/0 conn(0x7f6e2c0b4360 legacy=0x7f6e2c14d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:42.026 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.026+0000 7f6e3254d640 1 -- 172.21.15.196:0/4183442524 shutdown_connections 2024-03-29T10:40:42.026 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.026+0000 7f6e3254d640 1 -- 172.21.15.196:0/4183442524 >> 172.21.15.196:0/4183442524 conn(0x7f6e2c066b00 msgr2=0x7f6e2c066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:40:42.026 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.026+0000 7f6e3254d640 1 -- 172.21.15.196:0/4183442524 shutdown_connections 2024-03-29T10:40:42.026 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.026+0000 7f6e3254d640 1 -- 172.21.15.196:0/4183442524 wait complete. 2024-03-29T10:40:42.026 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.027+0000 7f6e3254d640 10 monclient: build_initial_monmap 2024-03-29T10:40:42.026 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.027+0000 7f6e3254d640 10 monclient: monmap: 2024-03-29T10:40:42.027 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 0 2024-03-29T10:40:42.027 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:40:42.027 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 0.000000 2024-03-29T10:40:42.027 INFO:teuthology.orchestra.run.smithi196.stderr:created 0.000000 2024-03-29T10:40:42.027 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 0 (unknown) 2024-03-29T10:40:42.027 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 1 2024-03-29T10:40:42.027 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-03-29T10:40:42.027 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-29T10:40:42.027 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.88:6790/0 mon.noname-c 2024-03-29T10:40:42.027 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-29T10:40:42.027 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.027+0000 7f6e3254d640 1 librados: starting msgr at 2024-03-29T10:40:42.027 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.027+0000 7f6e3254d640 1 librados: starting objecter 2024-03-29T10:40:42.028 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.027+0000 7f6e3254d640 1 Processor -- start 2024-03-29T10:40:42.028 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.027+0000 7f6e3254d640 1 -- start start 2024-03-29T10:40:42.028 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.027+0000 7f6e3254d640 1 librados: setting wanted keys 2024-03-29T10:40:42.028 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.027+0000 7f6e3254d640 1 librados: calling monclient init 2024-03-29T10:40:42.028 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.027+0000 7f6e3254d640 10 monclient: init 2024-03-29T10:40:42.028 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.028+0000 7f6e3254d640 10 monclient: _reopen_session rank -1 2024-03-29T10:40:42.028 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.028+0000 7f6e3254d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:40:42.028 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.028+0000 7f6e3254d640 10 monclient(hunting): picked mon.noname-a con 0x7f6e2c09ac40 addr v1:172.21.15.88:6789/0 2024-03-29T10:40:42.028 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.028+0000 7f6e3254d640 10 monclient(hunting): picked mon.noname-b con 0x7f6e2c0adff0 addr v1:172.21.15.196:6789/0 2024-03-29T10:40:42.028 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.028+0000 7f6e3254d640 10 monclient(hunting): picked mon.noname-c con 0x7f6e2c1462e0 addr v1:172.21.15.88:6790/0 2024-03-29T10:40:42.028 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.028+0000 7f6e3254d640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e2c140fb0 con 0x7f6e2c09ac40 2024-03-29T10:40:42.028 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.028+0000 7f6e3254d640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e2c1d74d0 con 0x7f6e2c0adff0 2024-03-29T10:40:42.028 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.028+0000 7f6e3254d640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e2c0b19d0 con 0x7f6e2c1462e0 2024-03-29T10:40:42.029 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.028+0000 7f6e3254d640 10 monclient(hunting): _renew_subs 2024-03-29T10:40:42.029 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.028+0000 7f6e3254d640 10 monclient(hunting): authenticate will time out at 788.343353s 2024-03-29T10:40:42.029 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.028+0000 7f6e2b7fe640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f6e2c0adff0 0x7f6e2c145bd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.196:39958/0 (socket says 172.21.15.196:39958) 2024-03-29T10:40:42.029 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.028+0000 7f6e2b7fe640 1 -- 172.21.15.196:0/2023328542 learned_addr learned my addr 172.21.15.196:0/2023328542 (peer_addr_for_me v1:172.21.15.196:0/0) 2024-03-29T10:40:42.029 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.028+0000 7f6e28ff9640 10 client.?.objecter ms_handle_connect 0x7f6e2c0adff0 2024-03-29T10:40:42.029 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.028+0000 7f6e28ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:40:42.029 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.028+0000 7f6e28ff9640 10 client.?.objecter ms_handle_connect 0x7f6e2c1462e0 2024-03-29T10:40:42.029 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.028+0000 7f6e28ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:40:42.029 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.028+0000 7f6e28ff9640 10 client.?.objecter ms_handle_connect 0x7f6e2c09ac40 2024-03-29T10:40:42.029 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.028+0000 7f6e28ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:40:42.029 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.028+0000 7f6e28ff9640 1 -- 172.21.15.196:0/2023328542 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4260045634 0 0) 0x7f6e2c1d74d0 con 0x7f6e2c0adff0 2024-03-29T10:40:42.029 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.028+0000 7f6e28ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:40:42.030 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.028+0000 7f6e28ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:42.030 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.028+0000 7f6e28ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:42.030 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.028+0000 7f6e28ff9640 10 monclient(hunting): my global_id is 4141 2024-03-29T10:40:42.030 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.028+0000 7f6e28ff9640 1 -- 172.21.15.196:0/2023328542 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e18004eb0 con 0x7f6e2c0adff0 2024-03-29T10:40:42.030 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.028+0000 7f6e28ff9640 1 -- 172.21.15.196:0/2023328542 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3523784473 0 0) 0x7f6e2c140fb0 con 0x7f6e2c09ac40 2024-03-29T10:40:42.030 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.028+0000 7f6e28ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:40:42.030 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.028+0000 7f6e28ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:42.030 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.028+0000 7f6e28ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:42.030 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.028+0000 7f6e28ff9640 10 monclient(hunting): my global_id is 4143 2024-03-29T10:40:42.030 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.028+0000 7f6e28ff9640 1 -- 172.21.15.196:0/2023328542 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e2c1d74d0 con 0x7f6e2c09ac40 2024-03-29T10:40:42.030 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.028+0000 7f6e28ff9640 1 -- 172.21.15.196:0/2023328542 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1025635599 0 0) 0x7f6e2c0b19d0 con 0x7f6e2c1462e0 2024-03-29T10:40:42.030 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.028+0000 7f6e28ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:40:42.031 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.028+0000 7f6e28ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:42.031 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.028+0000 7f6e28ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:42.031 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.028+0000 7f6e28ff9640 10 monclient(hunting): my global_id is 4139 2024-03-29T10:40:42.031 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.028+0000 7f6e28ff9640 1 -- 172.21.15.196:0/2023328542 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e2c140fb0 con 0x7f6e2c1462e0 2024-03-29T10:40:42.031 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.028+0000 7f6e28ff9640 1 -- 172.21.15.196:0/2023328542 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2680758630 0 0) 0x7f6e18004eb0 con 0x7f6e2c0adff0 2024-03-29T10:40:42.031 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.028+0000 7f6e28ff9640 1 -- 172.21.15.196:0/2023328542 --> v1:172.21.15.196:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6e2c0b19d0 con 0x7f6e2c0adff0 2024-03-29T10:40:42.031 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.028+0000 7f6e28ff9640 1 -- 172.21.15.196:0/2023328542 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3253417361 0 0) 0x7f6e2c1d74d0 con 0x7f6e2c09ac40 2024-03-29T10:40:42.031 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.028+0000 7f6e28ff9640 1 -- 172.21.15.196:0/2023328542 --> v1:172.21.15.88:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6e18004eb0 con 0x7f6e2c09ac40 2024-03-29T10:40:42.031 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.029+0000 7f6e28ff9640 1 -- 172.21.15.196:0/2023328542 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f6e1c001980 con 0x7f6e2c0adff0 2024-03-29T10:40:42.031 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.029+0000 7f6e28ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:42.031 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.029+0000 7f6e28ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:40:42.031 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.029+0000 7f6e28ff9640 10 monclient(hunting): dump: 2024-03-29T10:40:42.032 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 1 2024-03-29T10:40:42.032 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:40:42.032 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:40:42.032 INFO:teuthology.orchestra.run.smithi196.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:40:42.032 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:42.032 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 1 2024-03-29T10:40:42.032 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:40:42.032 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:40:42.032 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:40:42.032 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-29T10:40:42.032 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.029+0000 7f6e28ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:40:42.032 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.029+0000 7f6e28ff9640 1 -- 172.21.15.196:0/2023328542 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f6e20001890 con 0x7f6e2c09ac40 2024-03-29T10:40:42.033 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.029+0000 7f6e28ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:42.033 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.029+0000 7f6e28ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:40:42.033 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.029+0000 7f6e28ff9640 10 monclient(hunting): dump: 2024-03-29T10:40:42.033 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 1 2024-03-29T10:40:42.033 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:40:42.033 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:40:42.033 INFO:teuthology.orchestra.run.smithi196.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:40:42.033 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:42.033 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 1 2024-03-29T10:40:42.033 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:40:42.033 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:40:42.033 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:40:42.033 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-29T10:40:42.034 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.029+0000 7f6e28ff9640 1 -- 172.21.15.196:0/2023328542 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2420703624 0 0) 0x7f6e2c140fb0 con 0x7f6e2c1462e0 2024-03-29T10:40:42.034 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.029+0000 7f6e28ff9640 1 -- 172.21.15.196:0/2023328542 --> v1:172.21.15.88:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6e2c1d74d0 con 0x7f6e2c1462e0 2024-03-29T10:40:42.034 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.029+0000 7f6e28ff9640 1 -- 172.21.15.196:0/2023328542 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f6e24001d60 con 0x7f6e2c1462e0 2024-03-29T10:40:42.034 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.029+0000 7f6e28ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:42.034 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.029+0000 7f6e28ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:40:42.034 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.029+0000 7f6e28ff9640 10 monclient(hunting): dump: 2024-03-29T10:40:42.034 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 1 2024-03-29T10:40:42.035 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:40:42.035 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:40:42.035 INFO:teuthology.orchestra.run.smithi196.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:40:42.035 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:42.035 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 1 2024-03-29T10:40:42.035 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:40:42.035 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:40:42.035 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:40:42.035 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-29T10:40:42.035 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.029+0000 7f6e28ff9640 1 -- 172.21.15.196:0/2023328542 <== mon.1 v1:172.21.15.196:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3802350732 0 0) 0x7f6e2c0b19d0 con 0x7f6e2c0adff0 2024-03-29T10:40:42.035 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.029+0000 7f6e28ff9640 1 -- 172.21.15.196:0/2023328542 >> v1:172.21.15.88:6790/0 conn(0x7f6e2c1462e0 legacy=0x7f6e2c1e2c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:42.035 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.029+0000 7f6e28ff9640 1 -- 172.21.15.196:0/2023328542 >> v1:172.21.15.88:6789/0 conn(0x7f6e2c09ac40 legacy=0x7f6e2c1d79f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:42.036 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.029+0000 7f6e28ff9640 10 monclient: _finish_hunting 0 2024-03-29T10:40:42.036 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.029+0000 7f6e28ff9640 1 monclient: found mon.b 2024-03-29T10:40:42.036 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.029+0000 7f6e28ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:40:42.036 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.029+0000 7f6e28ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T10:40:42.036 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.029+0000 7f6e28ff9640 1 -- 172.21.15.196:0/2023328542 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e2c0b1b60 con 0x7f6e2c0adff0 2024-03-29T10:40:42.036 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.029+0000 7f6e28ff9640 10 monclient: _finish_auth 0 2024-03-29T10:40:42.036 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.029+0000 7f6e28ff9640 10 monclient: _check_auth_tickets 2024-03-29T10:40:42.036 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.029+0000 7f6e28ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:40:42.036 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.029+0000 7f6e3254d640 5 monclient: authenticate success, global_id 4141 2024-03-29T10:40:42.036 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.029+0000 7f6e3254d640 10 monclient: _renew_subs 2024-03-29T10:40:42.036 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.029+0000 7f6e3254d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T10:40:42.036 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.029+0000 7f6e3254d640 1 -- 172.21.15.196:0/2023328542 --> v1:172.21.15.196:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e2c1d8100 con 0x7f6e2c0adff0 2024-03-29T10:40:42.037 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.029+0000 7f6e3254d640 10 client.4141.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:40:42.037 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.029+0000 7f6e3254d640 10 monclient: _renew_subs 2024-03-29T10:40:42.037 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.029+0000 7f6e3254d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T10:40:42.037 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.029+0000 7f6e3254d640 1 -- 172.21.15.196:0/2023328542 --> v1:172.21.15.196:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e2c140180 con 0x7f6e2c0adff0 2024-03-29T10:40:42.037 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.029+0000 7f6e3254d640 1 librados: init done 2024-03-29T10:40:42.037 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.029+0000 7f6e28ff9640 1 -- 172.21.15.196:0/2023328542 <== mon.1 v1:172.21.15.196:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6e1c001ae0 con 0x7f6e2c0adff0 2024-03-29T10:40:42.037 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.029+0000 7f6e28ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:40:42.037 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.029+0000 7f6e28ff9640 1 -- 172.21.15.196:0/2023328542 <== mon.1 v1:172.21.15.196:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f6e1c0038c0 con 0x7f6e2c0adff0 2024-03-29T10:40:42.037 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.029+0000 7f6e28ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:42.037 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.029+0000 7f6e28ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:40:42.037 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.029+0000 7f6e2a7fc640 4 set_mon_vals no callback set 2024-03-29T10:40:42.037 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.029+0000 7f6e28ff9640 10 monclient: dump: 2024-03-29T10:40:42.037 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 1 2024-03-29T10:40:42.038 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:40:42.038 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:40:42.038 INFO:teuthology.orchestra.run.smithi196.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:40:42.038 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:42.038 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 1 2024-03-29T10:40:42.038 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:40:42.038 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:40:42.038 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:40:42.038 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-29T10:40:42.038 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.029+0000 7f6e28ff9640 1 -- 172.21.15.196:0/2023328542 <== mon.1 v1:172.21.15.196:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f6e1c003db0 con 0x7f6e2c0adff0 2024-03-29T10:40:42.038 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.029+0000 7f6e28ff9640 1 -- 172.21.15.196:0/2023328542 <== mon.1 v1:172.21.15.196:6789/0 8 ==== osd_map(7..7 src has 1..7) v4 ==== 1300+0+0 (unknown 2783968126 0 0) 0x7f6e1c0057c0 con 0x7f6e2c0adff0 2024-03-29T10:40:42.038 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.029+0000 7f6e28ff9640 10 client.4141.objecter ms_dispatch 0x7f6e2c000b90 osd_map(7..7 src has 1..7) v4 2024-03-29T10:40:42.039 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.029+0000 7f6e28ff9640 3 client.4141.objecter handle_osd_map got epochs [7,7] > 0 2024-03-29T10:40:42.039 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.029+0000 7f6e28ff9640 3 client.4141.objecter handle_osd_map decoding full epoch 7 2024-03-29T10:40:42.039 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.029+0000 7f6e28ff9640 20 client.4141.objecter dump_active .. 0 homeless 2024-03-29T10:40:42.039 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.030+0000 7f6e3254d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:40:42.039 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.030+0000 7f6e3254d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:40:42.039 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.030+0000 7f6e3254d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T10:40:42.039 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.030+0000 7f6e3254d640 1 -- 172.21.15.196:0/2023328542 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e0c005080 con 0x7f6e2c0adff0 2024-03-29T10:40:42.039 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.032+0000 7f6e28ff9640 1 -- 172.21.15.196:0/2023328542 <== mon.1 v1:172.21.15.196:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f6e1c005b60 con 0x7f6e2c0adff0 2024-03-29T10:40:42.039 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.032+0000 7f6e28ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:40:42.039 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.032+0000 7f6e28ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:40:42.086 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.102+0000 7f6e3254d640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "dc6d9911-785d-432f-8b53-56a1ccab302f", "id": 5}] 2024-03-29T10:40:42.086 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.102+0000 7f6e3254d640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "dc6d9911-785d-432f-8b53-56a1ccab302f", "id": 5}] 2024-03-29T10:40:42.086 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.102+0000 7f6e3254d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T10:40:42.086 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.102+0000 7f6e3254d640 1 -- 172.21.15.196:0/2023328542 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "osd new", "uuid": "dc6d9911-785d-432f-8b53-56a1ccab302f", "id": 5} v 0) v1 -- 0x7f6e0c0058b0 con 0x7f6e2c0adff0 2024-03-29T10:40:42.090 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.106+0000 7f6e28ff9640 1 -- 172.21.15.196:0/2023328542 <== mon.1 v1:172.21.15.196:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "dc6d9911-785d-432f-8b53-56a1ccab302f", "id": 5}]=0 v8) v1 ==== 112+0+2 (unknown 1142608415 0 1988412176) 0x7f6e1c004130 con 0x7f6e2c0adff0 2024-03-29T10:40:42.090 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.106+0000 7f6e28ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "dc6d9911-785d-432f-8b53-56a1ccab302f", "id": 5}] 2024-03-29T10:40:42.090 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.106+0000 7f6e28ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:40:42.090 INFO:teuthology.orchestra.run.smithi196.stdout:5 2024-03-29T10:40:42.092 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.107+0000 7f6e3254d640 10 librados: watch_flush enter 2024-03-29T10:40:42.092 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.107+0000 7f6e3254d640 10 librados: watch_flush exit 2024-03-29T10:40:42.092 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.107+0000 7f6e3254d640 20 client.4141.objecter shutdown clearing up homeless session... 2024-03-29T10:40:42.092 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.107+0000 7f6e3254d640 10 client.4141.objecter successfully canceled tick 2024-03-29T10:40:42.092 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.107+0000 7f6e3254d640 10 monclient: shutdown 2024-03-29T10:40:42.092 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.107+0000 7f6e3254d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:40:42.093 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.107+0000 7f6e3254d640 1 -- 172.21.15.196:0/2023328542 >> v1:172.21.15.196:6789/0 conn(0x7f6e2c0adff0 legacy=0x7f6e2c145bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:42.093 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.107+0000 7f6e3254d640 1 -- 172.21.15.196:0/2023328542 shutdown_connections 2024-03-29T10:40:42.093 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.107+0000 7f6e3254d640 1 -- 172.21.15.196:0/2023328542 >> 172.21.15.196:0/2023328542 conn(0x7f6e2c066b00 msgr2=0x7f6e2c13c350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:40:42.093 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.107+0000 7f6e3254d640 1 -- 172.21.15.196:0/2023328542 shutdown_connections 2024-03-29T10:40:42.093 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.107+0000 7f6e3254d640 1 -- 172.21.15.196:0/2023328542 wait complete. 2024-03-29T10:40:42.093 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.107+0000 7f6e3254d640 1 librados: shutdown 2024-03-29T10:40:42.101 DEBUG:teuthology.orchestra.run.smithi196:> sudo ceph --cluster ceph osd new 77c88437-b587-496e-8b4d-1824ea8b5305 6 2024-03-29T10:40:42.243 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.259+0000 7f92b1179640 10 monclient: get_monmap_and_config 2024-03-29T10:40:42.243 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.259+0000 7f92b1179640 10 monclient: build_initial_monmap 2024-03-29T10:40:42.243 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.259+0000 7f92b1179640 10 monclient: monmap: 2024-03-29T10:40:42.243 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 0 2024-03-29T10:40:42.243 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:40:42.243 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 2024-03-29T10:40:42.259992+0000 2024-03-29T10:40:42.243 INFO:teuthology.orchestra.run.smithi196.stderr:created 2024-03-29T10:40:42.259992+0000 2024-03-29T10:40:42.243 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 0 (unknown) 2024-03-29T10:40:42.243 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 1 2024-03-29T10:40:42.243 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-03-29T10:40:42.243 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-29T10:40:42.244 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.88:6790/0 mon.noname-b 2024-03-29T10:40:42.244 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-29T10:40:42.244 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.259+0000 7f92b1179640 1 Processor -- start 2024-03-29T10:40:42.244 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.259+0000 7f92b1179640 1 -- start start 2024-03-29T10:40:42.244 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.259+0000 7f92b1179640 10 monclient: init 2024-03-29T10:40:42.253 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.260+0000 7f92b1179640 10 monclient: _reopen_session rank -1 2024-03-29T10:40:42.253 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.260+0000 7f92b1179640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:40:42.253 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.260+0000 7f92b1179640 10 monclient(hunting): picked mon.noname-c con 0x7f92ac09ac40 addr v1:172.21.15.196:6789/0 2024-03-29T10:40:42.253 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.260+0000 7f92b1179640 10 monclient(hunting): picked mon.noname-a con 0x7f92ac09c870 addr v1:172.21.15.88:6789/0 2024-03-29T10:40:42.253 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.260+0000 7f92b1179640 10 monclient(hunting): picked mon.noname-b con 0x7f92ac0b4360 addr v1:172.21.15.88:6790/0 2024-03-29T10:40:42.253 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.260+0000 7f92b1179640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92ac09dec0 con 0x7f92ac09c870 2024-03-29T10:40:42.253 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.260+0000 7f92b1179640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92ac09da30 con 0x7f92ac09ac40 2024-03-29T10:40:42.253 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.260+0000 7f92b1179640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92ac0b39c0 con 0x7f92ac0b4360 2024-03-29T10:40:42.253 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.260+0000 7f92b1179640 10 monclient(hunting): _renew_subs 2024-03-29T10:40:42.254 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.260+0000 7f92b1179640 10 monclient(hunting): authenticate will time out at 788.575343s 2024-03-29T10:40:42.254 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.260+0000 7f92aad9d640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f92ac09ac40 0x7f92ac0b4a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.196:39970/0 (socket says 172.21.15.196:39970) 2024-03-29T10:40:42.254 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.260+0000 7f92aad9d640 1 -- 172.21.15.196:0/3660909205 learned_addr learned my addr 172.21.15.196:0/3660909205 (peer_addr_for_me v1:172.21.15.196:0/0) 2024-03-29T10:40:42.254 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.260+0000 7f92a9d9b640 1 -- 172.21.15.196:0/3660909205 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1008167562 0 0) 0x7f92ac09da30 con 0x7f92ac09ac40 2024-03-29T10:40:42.254 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.260+0000 7f92a9d9b640 10 monclient(hunting): _negotiate 2024-03-29T10:40:42.254 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.260+0000 7f92a9d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:42.254 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.260+0000 7f92a9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:42.254 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.260+0000 7f92a9d9b640 10 monclient(hunting): my global_id is 4144 2024-03-29T10:40:42.254 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.260+0000 7f92a9d9b640 1 -- 172.21.15.196:0/3660909205 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9294003520 con 0x7f92ac09ac40 2024-03-29T10:40:42.254 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.260+0000 7f92a9d9b640 1 -- 172.21.15.196:0/3660909205 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2855477007 0 0) 0x7f929c001040 con 0x7f92ac09c870 2024-03-29T10:40:42.254 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.260+0000 7f92a9d9b640 10 monclient(hunting): _negotiate 2024-03-29T10:40:42.254 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.260+0000 7f92a9d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:42.254 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.260+0000 7f92a9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:42.255 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.260+0000 7f92a9d9b640 10 monclient(hunting): my global_id is 4146 2024-03-29T10:40:42.255 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.260+0000 7f92a9d9b640 1 -- 172.21.15.196:0/3660909205 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f92ac09da30 con 0x7f92ac09c870 2024-03-29T10:40:42.255 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.260+0000 7f92a9d9b640 1 -- 172.21.15.196:0/3660909205 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1708024869 0 0) 0x7f92ac0b39c0 con 0x7f92ac0b4360 2024-03-29T10:40:42.255 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.260+0000 7f92a9d9b640 10 monclient(hunting): _negotiate 2024-03-29T10:40:42.255 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.260+0000 7f92a9d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:42.255 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.260+0000 7f92a9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:42.255 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.260+0000 7f92a9d9b640 10 monclient(hunting): my global_id is 4142 2024-03-29T10:40:42.255 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.260+0000 7f92a9d9b640 1 -- 172.21.15.196:0/3660909205 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f929c001040 con 0x7f92ac0b4360 2024-03-29T10:40:42.255 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.260+0000 7f92a9d9b640 1 -- 172.21.15.196:0/3660909205 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 641319559 0 0) 0x7f9294003520 con 0x7f92ac09ac40 2024-03-29T10:40:42.255 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.260+0000 7f92a9d9b640 1 -- 172.21.15.196:0/3660909205 >> v1:172.21.15.88:6790/0 conn(0x7f92ac0b4360 legacy=0x7f92ac14d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:42.255 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.260+0000 7f92a9d9b640 1 -- 172.21.15.196:0/3660909205 >> v1:172.21.15.88:6789/0 conn(0x7f92ac09c870 legacy=0x7f92ac13f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:42.255 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.260+0000 7f92a9d9b640 10 monclient: _finish_hunting 0 2024-03-29T10:40:42.255 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.260+0000 7f92a9d9b640 1 monclient: found mon.noname-c 2024-03-29T10:40:42.255 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.260+0000 7f92a9d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:40:42.256 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.260+0000 7f92a9d9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.196:6789/0 2024-03-29T10:40:42.256 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.260+0000 7f92a9d9b640 1 -- 172.21.15.196:0/3660909205 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92ac0b5160 con 0x7f92ac09ac40 2024-03-29T10:40:42.256 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.260+0000 7f92a9d9b640 10 monclient: _finish_auth 0 2024-03-29T10:40:42.256 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.260+0000 7f92a9d9b640 10 monclient: _check_auth_tickets 2024-03-29T10:40:42.256 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.260+0000 7f92a9d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:40:42.256 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.260+0000 7f92a9d9b640 1 -- 172.21.15.196:0/3660909205 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f92a0001690 con 0x7f92ac09ac40 2024-03-29T10:40:42.256 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.260+0000 7f92b1179640 5 monclient: authenticate success, global_id 4144 2024-03-29T10:40:42.256 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.260+0000 7f92b1179640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:40:42.256 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.260+0000 7f92a9d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:42.256 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.260+0000 7f92a9d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:40:42.256 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.260+0000 7f92a9d9b640 10 monclient: dump: 2024-03-29T10:40:42.256 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 1 2024-03-29T10:40:42.256 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:40:42.256 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:40:42.256 INFO:teuthology.orchestra.run.smithi196.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:40:42.257 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:42.257 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 1 2024-03-29T10:40:42.257 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:40:42.257 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:40:42.257 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:40:42.257 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-29T10:40:42.257 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.260+0000 7f92b1179640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:40:42.257 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.261+0000 7f92a9d9b640 1 -- 172.21.15.196:0/3660909205 <== mon.1 v1:172.21.15.196:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f92a0001870 con 0x7f92ac09ac40 2024-03-29T10:40:42.257 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.261+0000 7f92a9d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:40:42.257 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.261+0000 7f92a9d9b640 1 -- 172.21.15.196:0/3660909205 <== mon.1 v1:172.21.15.196:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f92a0001c80 con 0x7f92ac09ac40 2024-03-29T10:40:42.257 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.261+0000 7f92b1179640 10 monclient: get_monmap_and_config success 2024-03-29T10:40:42.257 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.261+0000 7f92b1179640 4 set_mon_vals no callback set 2024-03-29T10:40:42.257 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.261+0000 7f92a9d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:42.257 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.261+0000 7f92a9d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:40:42.258 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.261+0000 7f92a9d9b640 10 monclient: dump: 2024-03-29T10:40:42.258 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 1 2024-03-29T10:40:42.258 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:40:42.258 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:40:42.258 INFO:teuthology.orchestra.run.smithi196.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:40:42.258 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:42.258 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 1 2024-03-29T10:40:42.258 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:40:42.258 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:40:42.258 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:40:42.258 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-29T10:40:42.258 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.261+0000 7f92b1179640 10 monclient: shutdown 2024-03-29T10:40:42.258 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.261+0000 7f92b1179640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:40:42.259 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.261+0000 7f92b1179640 1 -- 172.21.15.196:0/3660909205 >> v1:172.21.15.196:6789/0 conn(0x7f92ac09ac40 legacy=0x7f92ac0b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:42.259 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.261+0000 7f92b1179640 1 -- 172.21.15.196:0/3660909205 shutdown_connections 2024-03-29T10:40:42.259 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.261+0000 7f92b1179640 1 -- 172.21.15.196:0/3660909205 >> 172.21.15.196:0/3660909205 conn(0x7f92ac066b00 msgr2=0x7f92ac066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:40:42.259 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.261+0000 7f92b1179640 1 -- 172.21.15.196:0/3660909205 shutdown_connections 2024-03-29T10:40:42.259 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.261+0000 7f92b1179640 1 -- 172.21.15.196:0/3660909205 wait complete. 2024-03-29T10:40:42.259 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.262+0000 7f92b1179640 10 monclient: build_initial_monmap 2024-03-29T10:40:42.259 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.262+0000 7f92b1179640 10 monclient: monmap: 2024-03-29T10:40:42.259 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 0 2024-03-29T10:40:42.259 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:40:42.259 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 0.000000 2024-03-29T10:40:42.259 INFO:teuthology.orchestra.run.smithi196.stderr:created 0.000000 2024-03-29T10:40:42.259 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 0 (unknown) 2024-03-29T10:40:42.259 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 1 2024-03-29T10:40:42.259 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-03-29T10:40:42.260 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-29T10:40:42.260 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.88:6790/0 mon.noname-c 2024-03-29T10:40:42.260 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-29T10:40:42.260 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.262+0000 7f92b1179640 1 librados: starting msgr at 2024-03-29T10:40:42.260 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.262+0000 7f92b1179640 1 librados: starting objecter 2024-03-29T10:40:42.260 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.262+0000 7f92b1179640 1 Processor -- start 2024-03-29T10:40:42.260 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.262+0000 7f92b1179640 1 -- start start 2024-03-29T10:40:42.260 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.262+0000 7f92b1179640 1 librados: setting wanted keys 2024-03-29T10:40:42.260 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.262+0000 7f92b1179640 1 librados: calling monclient init 2024-03-29T10:40:42.260 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.262+0000 7f92b1179640 10 monclient: init 2024-03-29T10:40:42.260 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.263+0000 7f92b1179640 10 monclient: _reopen_session rank -1 2024-03-29T10:40:42.260 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.263+0000 7f92b1179640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:40:42.260 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.263+0000 7f92b1179640 10 monclient(hunting): picked mon.noname-a con 0x7f92ac09ac40 addr v1:172.21.15.88:6789/0 2024-03-29T10:40:42.261 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.263+0000 7f92b1179640 10 monclient(hunting): picked mon.noname-b con 0x7f92ac1435c0 addr v1:172.21.15.196:6789/0 2024-03-29T10:40:42.261 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.263+0000 7f92b1179640 10 monclient(hunting): picked mon.noname-c con 0x7f92ac1dab60 addr v1:172.21.15.88:6790/0 2024-03-29T10:40:42.261 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.263+0000 7f92b1179640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92ac146040 con 0x7f92ac09ac40 2024-03-29T10:40:42.261 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.263+0000 7f92b1179640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92ac1462e0 con 0x7f92ac1435c0 2024-03-29T10:40:42.261 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.263+0000 7f92b1179640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92ac144a70 con 0x7f92ac1dab60 2024-03-29T10:40:42.261 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.263+0000 7f92b1179640 10 monclient(hunting): _renew_subs 2024-03-29T10:40:42.262 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.263+0000 7f92b1179640 10 monclient(hunting): authenticate will time out at 788.578463s 2024-03-29T10:40:42.262 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.263+0000 7f92aa59c640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f92ac1435c0 0x7f92ac1da450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.196:39978/0 (socket says 172.21.15.196:39978) 2024-03-29T10:40:42.262 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.263+0000 7f92aa59c640 1 -- 172.21.15.196:0/4222659304 learned_addr learned my addr 172.21.15.196:0/4222659304 (peer_addr_for_me v1:172.21.15.196:0/0) 2024-03-29T10:40:42.262 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.263+0000 7f92937fe640 10 client.?.objecter ms_handle_connect 0x7f92ac1435c0 2024-03-29T10:40:42.262 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.263+0000 7f92937fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:40:42.262 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.263+0000 7f92937fe640 10 client.?.objecter ms_handle_connect 0x7f92ac09ac40 2024-03-29T10:40:42.262 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.263+0000 7f92937fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:40:42.262 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.263+0000 7f92937fe640 10 client.?.objecter ms_handle_connect 0x7f92ac1dab60 2024-03-29T10:40:42.262 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.263+0000 7f92937fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:40:42.262 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.263+0000 7f92937fe640 1 -- 172.21.15.196:0/4222659304 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2628329256 0 0) 0x7f92ac1462e0 con 0x7f92ac1435c0 2024-03-29T10:40:42.262 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.263+0000 7f92937fe640 10 monclient(hunting): _negotiate 2024-03-29T10:40:42.262 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.263+0000 7f92937fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:42.262 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.263+0000 7f92937fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:42.262 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.263+0000 7f92937fe640 10 monclient(hunting): my global_id is 4147 2024-03-29T10:40:42.263 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.263+0000 7f92937fe640 1 -- 172.21.15.196:0/4222659304 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9298004eb0 con 0x7f92ac1435c0 2024-03-29T10:40:42.263 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.263+0000 7f92937fe640 1 -- 172.21.15.196:0/4222659304 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1329522698 0 0) 0x7f92ac146040 con 0x7f92ac09ac40 2024-03-29T10:40:42.263 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.263+0000 7f92937fe640 10 monclient(hunting): _negotiate 2024-03-29T10:40:42.263 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.263+0000 7f92937fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:42.263 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.263+0000 7f92937fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:42.263 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.263+0000 7f92937fe640 10 monclient(hunting): my global_id is 4149 2024-03-29T10:40:42.263 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.263+0000 7f92937fe640 1 -- 172.21.15.196:0/4222659304 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f92ac1462e0 con 0x7f92ac09ac40 2024-03-29T10:40:42.263 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.263+0000 7f92937fe640 1 -- 172.21.15.196:0/4222659304 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 358854669 0 0) 0x7f92ac144a70 con 0x7f92ac1dab60 2024-03-29T10:40:42.263 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.263+0000 7f92937fe640 10 monclient(hunting): _negotiate 2024-03-29T10:40:42.263 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.263+0000 7f92937fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:42.263 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.263+0000 7f92937fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:42.263 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.263+0000 7f92937fe640 10 monclient(hunting): my global_id is 4145 2024-03-29T10:40:42.263 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.263+0000 7f92937fe640 1 -- 172.21.15.196:0/4222659304 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f92ac146040 con 0x7f92ac1dab60 2024-03-29T10:40:42.264 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.264+0000 7f92937fe640 1 -- 172.21.15.196:0/4222659304 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4262819886 0 0) 0x7f9298004eb0 con 0x7f92ac1435c0 2024-03-29T10:40:42.264 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.264+0000 7f92937fe640 1 -- 172.21.15.196:0/4222659304 --> v1:172.21.15.196:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f92ac144a70 con 0x7f92ac1435c0 2024-03-29T10:40:42.264 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.264+0000 7f92937fe640 1 -- 172.21.15.196:0/4222659304 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2662001809 0 0) 0x7f92ac1462e0 con 0x7f92ac09ac40 2024-03-29T10:40:42.264 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.264+0000 7f92937fe640 1 -- 172.21.15.196:0/4222659304 --> v1:172.21.15.88:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9298004eb0 con 0x7f92ac09ac40 2024-03-29T10:40:42.264 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.264+0000 7f92937fe640 1 -- 172.21.15.196:0/4222659304 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f929c001950 con 0x7f92ac1435c0 2024-03-29T10:40:42.264 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.264+0000 7f92937fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:42.264 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.264+0000 7f92937fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:40:42.264 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.264+0000 7f92937fe640 10 monclient(hunting): dump: 2024-03-29T10:40:42.264 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 1 2024-03-29T10:40:42.264 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:40:42.264 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:40:42.264 INFO:teuthology.orchestra.run.smithi196.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:40:42.264 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:42.264 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 1 2024-03-29T10:40:42.265 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:40:42.265 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:40:42.265 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:40:42.265 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-29T10:40:42.265 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.264+0000 7f92937fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:40:42.265 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.264+0000 7f92937fe640 1 -- 172.21.15.196:0/4222659304 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 992584675 0 0) 0x7f92ac146040 con 0x7f92ac1dab60 2024-03-29T10:40:42.265 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.264+0000 7f92937fe640 1 -- 172.21.15.196:0/4222659304 --> v1:172.21.15.88:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f92ac1462e0 con 0x7f92ac1dab60 2024-03-29T10:40:42.265 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.264+0000 7f92937fe640 1 -- 172.21.15.196:0/4222659304 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f92a0001b80 con 0x7f92ac09ac40 2024-03-29T10:40:42.265 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.264+0000 7f92937fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:42.265 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.264+0000 7f92937fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:40:42.265 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.264+0000 7f92937fe640 10 monclient(hunting): dump: 2024-03-29T10:40:42.265 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 1 2024-03-29T10:40:42.265 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:40:42.266 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:40:42.266 INFO:teuthology.orchestra.run.smithi196.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:40:42.266 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:42.266 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 1 2024-03-29T10:40:42.266 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:40:42.266 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:40:42.266 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:40:42.266 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-29T10:40:42.266 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.264+0000 7f92937fe640 1 -- 172.21.15.196:0/4222659304 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f92a4001b50 con 0x7f92ac1dab60 2024-03-29T10:40:42.266 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.264+0000 7f92937fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:42.266 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.264+0000 7f92937fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:40:42.266 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.264+0000 7f92937fe640 10 monclient(hunting): dump: 2024-03-29T10:40:42.266 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 1 2024-03-29T10:40:42.267 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:40:42.267 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:40:42.267 INFO:teuthology.orchestra.run.smithi196.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:40:42.267 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:42.267 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 1 2024-03-29T10:40:42.267 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:40:42.267 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:40:42.267 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:40:42.267 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-29T10:40:42.267 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.264+0000 7f92937fe640 1 -- 172.21.15.196:0/4222659304 <== mon.0 v1:172.21.15.88:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3549246300 0 0) 0x7f9298004eb0 con 0x7f92ac09ac40 2024-03-29T10:40:42.267 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.264+0000 7f92937fe640 1 -- 172.21.15.196:0/4222659304 >> v1:172.21.15.88:6790/0 conn(0x7f92ac1dab60 legacy=0x7f92ac1e18f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:42.267 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.264+0000 7f92937fe640 1 -- 172.21.15.196:0/4222659304 >> v1:172.21.15.196:6789/0 conn(0x7f92ac1435c0 legacy=0x7f92ac1da450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:42.268 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.264+0000 7f92937fe640 10 monclient: _finish_hunting 0 2024-03-29T10:40:42.268 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.264+0000 7f92937fe640 1 monclient: found mon.a 2024-03-29T10:40:42.268 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.264+0000 7f92937fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:40:42.268 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.264+0000 7f92937fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-03-29T10:40:42.268 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.264+0000 7f92937fe640 1 -- 172.21.15.196:0/4222659304 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92ac144c00 con 0x7f92ac09ac40 2024-03-29T10:40:42.268 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.264+0000 7f92937fe640 10 monclient: _finish_auth 0 2024-03-29T10:40:42.268 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.264+0000 7f92937fe640 10 monclient: _check_auth_tickets 2024-03-29T10:40:42.268 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.264+0000 7f92937fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:40:42.268 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.264+0000 7f92b1179640 5 monclient: authenticate success, global_id 4149 2024-03-29T10:40:42.268 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.264+0000 7f92b1179640 10 monclient: _renew_subs 2024-03-29T10:40:42.268 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.264+0000 7f92b1179640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-03-29T10:40:42.268 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.264+0000 7f92b1179640 1 -- 172.21.15.196:0/4222659304 --> v1:172.21.15.88:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92ac1dafb0 con 0x7f92ac09ac40 2024-03-29T10:40:42.269 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.264+0000 7f92b1179640 10 client.4149.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:40:42.269 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.264+0000 7f92b1179640 10 monclient: _renew_subs 2024-03-29T10:40:42.269 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.264+0000 7f92b1179640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-03-29T10:40:42.269 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.264+0000 7f92b1179640 1 -- 172.21.15.196:0/4222659304 --> v1:172.21.15.88:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f92ac1e31f0 con 0x7f92ac09ac40 2024-03-29T10:40:42.269 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.264+0000 7f92b1179640 1 librados: init done 2024-03-29T10:40:42.269 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.264+0000 7f92937fe640 1 -- 172.21.15.196:0/4222659304 <== mon.0 v1:172.21.15.88:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f92a0004bf0 con 0x7f92ac09ac40 2024-03-29T10:40:42.269 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.264+0000 7f92937fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:40:42.269 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.264+0000 7f92937fe640 1 -- 172.21.15.196:0/4222659304 <== mon.0 v1:172.21.15.88:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f92a0004f00 con 0x7f92ac09ac40 2024-03-29T10:40:42.269 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.264+0000 7f92937fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:42.269 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.264+0000 7f92937fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:40:42.269 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.264+0000 7f92937fe640 10 monclient: dump: 2024-03-29T10:40:42.269 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 1 2024-03-29T10:40:42.269 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:40:42.269 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:40:42.270 INFO:teuthology.orchestra.run.smithi196.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:40:42.270 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:42.270 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 1 2024-03-29T10:40:42.270 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:40:42.270 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:40:42.270 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:40:42.270 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-29T10:40:42.270 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.264+0000 7f92a959a640 4 set_mon_vals no callback set 2024-03-29T10:40:42.270 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.264+0000 7f92937fe640 1 -- 172.21.15.196:0/4222659304 <== mon.0 v1:172.21.15.88:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f92a00053f0 con 0x7f92ac09ac40 2024-03-29T10:40:42.270 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.264+0000 7f92937fe640 1 -- 172.21.15.196:0/4222659304 <== mon.0 v1:172.21.15.88:6789/0 8 ==== osd_map(8..8 src has 1..8) v4 ==== 1415+0+0 (unknown 2955403456 0 0) 0x7f92a0006a30 con 0x7f92ac09ac40 2024-03-29T10:40:42.270 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.264+0000 7f92937fe640 10 client.4149.objecter ms_dispatch 0x7f92ac000b90 osd_map(8..8 src has 1..8) v4 2024-03-29T10:40:42.270 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.264+0000 7f92937fe640 3 client.4149.objecter handle_osd_map got epochs [8,8] > 0 2024-03-29T10:40:42.270 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.264+0000 7f92937fe640 3 client.4149.objecter handle_osd_map decoding full epoch 8 2024-03-29T10:40:42.270 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.264+0000 7f92937fe640 20 client.4149.objecter dump_active .. 0 homeless 2024-03-29T10:40:42.271 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.265+0000 7f92b1179640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:40:42.271 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.265+0000 7f92b1179640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:40:42.271 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.265+0000 7f92b1179640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-03-29T10:40:42.271 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.265+0000 7f92b1179640 1 -- 172.21.15.196:0/4222659304 --> v1:172.21.15.88:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f928c005080 con 0x7f92ac09ac40 2024-03-29T10:40:42.271 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.267+0000 7f92937fe640 1 -- 172.21.15.196:0/4222659304 <== mon.0 v1:172.21.15.88:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f92a00064f0 con 0x7f92ac09ac40 2024-03-29T10:40:42.271 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.267+0000 7f92937fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:40:42.271 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.267+0000 7f92937fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:40:42.304 INFO:tasks.ceph.mgr.y.smithi088.stderr:2024-03-29T10:40:42.320+0000 7fca56440040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-29T10:40:42.322 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.338+0000 7f92b1179640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "77c88437-b587-496e-8b4d-1824ea8b5305", "id": 6}] 2024-03-29T10:40:42.322 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.338+0000 7f92b1179640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "77c88437-b587-496e-8b4d-1824ea8b5305", "id": 6}] 2024-03-29T10:40:42.322 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.338+0000 7f92b1179640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-03-29T10:40:42.322 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.338+0000 7f92b1179640 1 -- 172.21.15.196:0/4222659304 --> v1:172.21.15.88:6789/0 -- mon_command({"prefix": "osd new", "uuid": "77c88437-b587-496e-8b4d-1824ea8b5305", "id": 6} v 0) v1 -- 0x7f928c0058b0 con 0x7f92ac09ac40 2024-03-29T10:40:42.325 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.340+0000 7f92937fe640 1 -- 172.21.15.196:0/4222659304 <== mon.0 v1:172.21.15.88:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "77c88437-b587-496e-8b4d-1824ea8b5305", "id": 6}]=0 v9) v1 ==== 112+0+2 (unknown 3058581815 0 1113811849) 0x7f92a0006760 con 0x7f92ac09ac40 2024-03-29T10:40:42.325 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.340+0000 7f92937fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "77c88437-b587-496e-8b4d-1824ea8b5305", "id": 6}] 2024-03-29T10:40:42.325 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.340+0000 7f92937fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:40:42.325 INFO:teuthology.orchestra.run.smithi196.stdout:6 2024-03-29T10:40:42.326 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.341+0000 7f92b1179640 10 librados: watch_flush enter 2024-03-29T10:40:42.326 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.341+0000 7f92b1179640 10 librados: watch_flush exit 2024-03-29T10:40:42.327 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.341+0000 7f92b1179640 20 client.4149.objecter shutdown clearing up homeless session... 2024-03-29T10:40:42.327 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.341+0000 7f92b1179640 10 client.4149.objecter successfully canceled tick 2024-03-29T10:40:42.327 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.341+0000 7f92b1179640 10 monclient: shutdown 2024-03-29T10:40:42.327 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.341+0000 7f92b1179640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:40:42.327 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.341+0000 7f92b1179640 1 -- 172.21.15.196:0/4222659304 >> v1:172.21.15.88:6789/0 conn(0x7f92ac09ac40 legacy=0x7f92ac144360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:42.327 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.341+0000 7f92b1179640 1 -- 172.21.15.196:0/4222659304 shutdown_connections 2024-03-29T10:40:42.327 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.341+0000 7f92b1179640 1 -- 172.21.15.196:0/4222659304 >> 172.21.15.196:0/4222659304 conn(0x7f92ac066b00 msgr2=0x7f92ac066ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:40:42.327 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.341+0000 7f92b1179640 1 -- 172.21.15.196:0/4222659304 shutdown_connections 2024-03-29T10:40:42.327 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.341+0000 7f92b1179640 1 -- 172.21.15.196:0/4222659304 wait complete. 2024-03-29T10:40:42.327 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.341+0000 7f92b1179640 1 librados: shutdown 2024-03-29T10:40:42.336 DEBUG:teuthology.orchestra.run.smithi196:> sudo ceph --cluster ceph osd new ce963d96-5a8e-44c3-99de-c0dbebe234ed 7 2024-03-29T10:40:42.341 INFO:tasks.ceph.mgr.x.smithi196.stderr:2024-03-29T10:40:42.358+0000 7ff5dcc4b040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-29T10:40:42.443 INFO:tasks.ceph.mgr.y.smithi088.stderr:2024-03-29T10:40:42.455+0000 7fca56440040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-29T10:40:42.449 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.452+0000 7fd928ae0640 10 monclient: get_monmap_and_config 2024-03-29T10:40:42.449 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.452+0000 7fd928ae0640 10 monclient: build_initial_monmap 2024-03-29T10:40:42.449 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.452+0000 7fd928ae0640 10 monclient: monmap: 2024-03-29T10:40:42.449 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 0 2024-03-29T10:40:42.449 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:40:42.449 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 2024-03-29T10:40:42.452937+0000 2024-03-29T10:40:42.450 INFO:teuthology.orchestra.run.smithi196.stderr:created 2024-03-29T10:40:42.452937+0000 2024-03-29T10:40:42.450 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 0 (unknown) 2024-03-29T10:40:42.450 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 1 2024-03-29T10:40:42.450 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-03-29T10:40:42.450 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-29T10:40:42.450 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.88:6790/0 mon.noname-b 2024-03-29T10:40:42.450 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-29T10:40:42.450 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.452+0000 7fd928ae0640 1 Processor -- start 2024-03-29T10:40:42.450 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.452+0000 7fd928ae0640 1 -- start start 2024-03-29T10:40:42.450 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.452+0000 7fd928ae0640 10 monclient: init 2024-03-29T10:40:42.450 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.453+0000 7fd928ae0640 10 monclient: _reopen_session rank -1 2024-03-29T10:40:42.450 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.453+0000 7fd928ae0640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:40:42.450 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.453+0000 7fd928ae0640 10 monclient(hunting): picked mon.noname-b con 0x7fd92409ac40 addr v1:172.21.15.88:6790/0 2024-03-29T10:40:42.451 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.453+0000 7fd928ae0640 10 monclient(hunting): picked mon.noname-a con 0x7fd92409c870 addr v1:172.21.15.88:6789/0 2024-03-29T10:40:42.451 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.453+0000 7fd928ae0640 10 monclient(hunting): picked mon.noname-c con 0x7fd9240b4360 addr v1:172.21.15.196:6789/0 2024-03-29T10:40:42.451 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.453+0000 7fd928ae0640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd92409dec0 con 0x7fd92409c870 2024-03-29T10:40:42.451 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.453+0000 7fd928ae0640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd92409da30 con 0x7fd9240b4360 2024-03-29T10:40:42.451 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.453+0000 7fd928ae0640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9240b39c0 con 0x7fd92409ac40 2024-03-29T10:40:42.451 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.453+0000 7fd928ae0640 10 monclient(hunting): _renew_subs 2024-03-29T10:40:42.451 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.453+0000 7fd928ae0640 10 monclient(hunting): authenticate will time out at 788.768395s 2024-03-29T10:40:42.451 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.453+0000 7fd922d9d640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7fd9240b4360 0x7fd92414d9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.196:39990/0 (socket says 172.21.15.196:39990) 2024-03-29T10:40:42.451 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.453+0000 7fd922d9d640 1 -- 172.21.15.196:0/3200876112 learned_addr learned my addr 172.21.15.196:0/3200876112 (peer_addr_for_me v1:172.21.15.196:0/0) 2024-03-29T10:40:42.451 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.453+0000 7fd921d9b640 1 -- 172.21.15.196:0/3200876112 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 889892974 0 0) 0x7fd92409da30 con 0x7fd9240b4360 2024-03-29T10:40:42.451 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.453+0000 7fd921d9b640 10 monclient(hunting): _negotiate 2024-03-29T10:40:42.451 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.453+0000 7fd921d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:42.451 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.453+0000 7fd921d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:42.451 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.453+0000 7fd921d9b640 10 monclient(hunting): my global_id is 4150 2024-03-29T10:40:42.452 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.453+0000 7fd921d9b640 1 -- 172.21.15.196:0/3200876112 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd910003520 con 0x7fd9240b4360 2024-03-29T10:40:42.452 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.453+0000 7fd921d9b640 1 -- 172.21.15.196:0/3200876112 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3274147445 0 0) 0x7fd9240b39c0 con 0x7fd92409ac40 2024-03-29T10:40:42.452 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.453+0000 7fd921d9b640 10 monclient(hunting): _negotiate 2024-03-29T10:40:42.452 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.453+0000 7fd921d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:42.452 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.453+0000 7fd921d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:42.452 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.453+0000 7fd921d9b640 10 monclient(hunting): my global_id is 4148 2024-03-29T10:40:42.452 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.453+0000 7fd921d9b640 1 -- 172.21.15.196:0/3200876112 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd92409da30 con 0x7fd92409ac40 2024-03-29T10:40:42.452 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.453+0000 7fd921d9b640 1 -- 172.21.15.196:0/3200876112 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3246160002 0 0) 0x7fd90c001110 con 0x7fd92409c870 2024-03-29T10:40:42.452 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.453+0000 7fd921d9b640 10 monclient(hunting): _negotiate 2024-03-29T10:40:42.452 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.453+0000 7fd921d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:42.452 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.453+0000 7fd921d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:42.452 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.453+0000 7fd921d9b640 10 monclient(hunting): my global_id is 4152 2024-03-29T10:40:42.452 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.453+0000 7fd921d9b640 1 -- 172.21.15.196:0/3200876112 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd9240b39c0 con 0x7fd92409c870 2024-03-29T10:40:42.453 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.454+0000 7fd921d9b640 1 -- 172.21.15.196:0/3200876112 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 297703200 0 0) 0x7fd910003520 con 0x7fd9240b4360 2024-03-29T10:40:42.453 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.454+0000 7fd921d9b640 1 -- 172.21.15.196:0/3200876112 >> v1:172.21.15.88:6790/0 conn(0x7fd92409ac40 legacy=0x7fd9240b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:42.453 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.454+0000 7fd921d9b640 1 -- 172.21.15.196:0/3200876112 >> v1:172.21.15.88:6789/0 conn(0x7fd92409c870 legacy=0x7fd92413f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:42.453 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.454+0000 7fd921d9b640 10 monclient: _finish_hunting 0 2024-03-29T10:40:42.453 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.454+0000 7fd921d9b640 1 monclient: found mon.noname-c 2024-03-29T10:40:42.453 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.454+0000 7fd921d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:40:42.453 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.454+0000 7fd921d9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.196:6789/0 2024-03-29T10:40:42.453 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.454+0000 7fd921d9b640 1 -- 172.21.15.196:0/3200876112 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9240b5160 con 0x7fd9240b4360 2024-03-29T10:40:42.453 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.454+0000 7fd921d9b640 10 monclient: _finish_auth 0 2024-03-29T10:40:42.453 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.454+0000 7fd921d9b640 10 monclient: _check_auth_tickets 2024-03-29T10:40:42.453 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.454+0000 7fd921d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:40:42.453 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.454+0000 7fd928ae0640 5 monclient: authenticate success, global_id 4150 2024-03-29T10:40:42.453 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.454+0000 7fd928ae0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:40:42.453 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.454+0000 7fd921d9b640 1 -- 172.21.15.196:0/3200876112 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7fd91c001a10 con 0x7fd9240b4360 2024-03-29T10:40:42.454 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.454+0000 7fd921d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:42.454 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.454+0000 7fd921d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:40:42.454 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.454+0000 7fd921d9b640 10 monclient: dump: 2024-03-29T10:40:42.454 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 1 2024-03-29T10:40:42.454 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:40:42.454 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:40:42.454 INFO:teuthology.orchestra.run.smithi196.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:40:42.454 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:42.454 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 1 2024-03-29T10:40:42.454 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:40:42.454 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:40:42.454 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:40:42.454 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-29T10:40:42.454 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.454+0000 7fd928ae0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:40:42.455 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.454+0000 7fd921d9b640 1 -- 172.21.15.196:0/3200876112 <== mon.1 v1:172.21.15.196:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd91c001bf0 con 0x7fd9240b4360 2024-03-29T10:40:42.455 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.454+0000 7fd921d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:40:42.455 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.454+0000 7fd928ae0640 10 monclient: get_monmap_and_config success 2024-03-29T10:40:42.455 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.454+0000 7fd921d9b640 1 -- 172.21.15.196:0/3200876112 <== mon.1 v1:172.21.15.196:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7fd91c002000 con 0x7fd9240b4360 2024-03-29T10:40:42.455 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.454+0000 7fd928ae0640 4 set_mon_vals no callback set 2024-03-29T10:40:42.455 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.454+0000 7fd921d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:42.455 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.454+0000 7fd921d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:40:42.455 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.454+0000 7fd921d9b640 10 monclient: dump: 2024-03-29T10:40:42.455 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 1 2024-03-29T10:40:42.455 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:40:42.455 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:40:42.455 INFO:teuthology.orchestra.run.smithi196.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:40:42.455 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:42.455 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 1 2024-03-29T10:40:42.456 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:40:42.456 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:40:42.456 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:40:42.456 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-29T10:40:42.456 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.454+0000 7fd928ae0640 10 monclient: shutdown 2024-03-29T10:40:42.456 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.454+0000 7fd928ae0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:40:42.456 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.454+0000 7fd928ae0640 1 -- 172.21.15.196:0/3200876112 >> v1:172.21.15.196:6789/0 conn(0x7fd9240b4360 legacy=0x7fd92414d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:42.456 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.454+0000 7fd928ae0640 1 -- 172.21.15.196:0/3200876112 shutdown_connections 2024-03-29T10:40:42.456 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.454+0000 7fd928ae0640 1 -- 172.21.15.196:0/3200876112 >> 172.21.15.196:0/3200876112 conn(0x7fd924066b00 msgr2=0x7fd924066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:40:42.456 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.454+0000 7fd928ae0640 1 -- 172.21.15.196:0/3200876112 shutdown_connections 2024-03-29T10:40:42.456 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.454+0000 7fd928ae0640 1 -- 172.21.15.196:0/3200876112 wait complete. 2024-03-29T10:40:42.456 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.455+0000 7fd928ae0640 10 monclient: build_initial_monmap 2024-03-29T10:40:42.456 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.455+0000 7fd928ae0640 10 monclient: monmap: 2024-03-29T10:40:42.456 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 0 2024-03-29T10:40:42.456 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:40:42.457 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 0.000000 2024-03-29T10:40:42.457 INFO:teuthology.orchestra.run.smithi196.stderr:created 0.000000 2024-03-29T10:40:42.457 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 0 (unknown) 2024-03-29T10:40:42.457 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 1 2024-03-29T10:40:42.457 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-03-29T10:40:42.457 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-29T10:40:42.457 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.88:6790/0 mon.noname-c 2024-03-29T10:40:42.457 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-29T10:40:42.457 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.455+0000 7fd928ae0640 1 librados: starting msgr at 2024-03-29T10:40:42.457 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.455+0000 7fd928ae0640 1 librados: starting objecter 2024-03-29T10:40:42.457 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.455+0000 7fd928ae0640 1 Processor -- start 2024-03-29T10:40:42.457 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.455+0000 7fd928ae0640 1 -- start start 2024-03-29T10:40:42.457 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.455+0000 7fd928ae0640 1 librados: setting wanted keys 2024-03-29T10:40:42.457 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.455+0000 7fd928ae0640 1 librados: calling monclient init 2024-03-29T10:40:42.457 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.455+0000 7fd928ae0640 10 monclient: init 2024-03-29T10:40:42.458 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.456+0000 7fd928ae0640 10 monclient: _reopen_session rank -1 2024-03-29T10:40:42.458 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.456+0000 7fd928ae0640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:40:42.458 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.456+0000 7fd928ae0640 10 monclient(hunting): picked mon.noname-a con 0x7fd92409ac40 addr v1:172.21.15.88:6789/0 2024-03-29T10:40:42.458 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.456+0000 7fd928ae0640 10 monclient(hunting): picked mon.noname-c con 0x7fd9240b7310 addr v1:172.21.15.88:6790/0 2024-03-29T10:40:42.458 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.456+0000 7fd928ae0640 10 monclient(hunting): picked mon.noname-b con 0x7fd9241d7de0 addr v1:172.21.15.196:6789/0 2024-03-29T10:40:42.458 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.456+0000 7fd928ae0640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9240b78c0 con 0x7fd92409ac40 2024-03-29T10:40:42.458 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.456+0000 7fd928ae0640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9240b7d50 con 0x7fd9241d7de0 2024-03-29T10:40:42.458 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.456+0000 7fd928ae0640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd924146480 con 0x7fd9240b7310 2024-03-29T10:40:42.458 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.456+0000 7fd928ae0640 10 monclient(hunting): _renew_subs 2024-03-29T10:40:42.458 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.456+0000 7fd928ae0640 10 monclient(hunting): authenticate will time out at 788.771479s 2024-03-29T10:40:42.458 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.456+0000 7fd922d9d640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7fd9241d7de0 0x7fd9241e2180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.196:39994/0 (socket says 172.21.15.196:39994) 2024-03-29T10:40:42.459 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.456+0000 7fd922d9d640 1 -- 172.21.15.196:0/4014296949 learned_addr learned my addr 172.21.15.196:0/4014296949 (peer_addr_for_me v1:172.21.15.196:0/0) 2024-03-29T10:40:42.459 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.456+0000 7fd91affd640 10 client.?.objecter ms_handle_connect 0x7fd9241d7de0 2024-03-29T10:40:42.459 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.456+0000 7fd91affd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:40:42.459 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.456+0000 7fd91affd640 10 client.?.objecter ms_handle_connect 0x7fd92409ac40 2024-03-29T10:40:42.459 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.456+0000 7fd91affd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:40:42.459 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.456+0000 7fd91affd640 1 -- 172.21.15.196:0/4014296949 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4089708404 0 0) 0x7fd9240b7d50 con 0x7fd9241d7de0 2024-03-29T10:40:42.459 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.456+0000 7fd91affd640 10 monclient(hunting): _negotiate 2024-03-29T10:40:42.459 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.456+0000 7fd91affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:42.459 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.456+0000 7fd91affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:42.459 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.456+0000 7fd91affd640 10 monclient(hunting): my global_id is 4153 2024-03-29T10:40:42.459 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.456+0000 7fd91affd640 1 -- 172.21.15.196:0/4014296949 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd904004eb0 con 0x7fd9241d7de0 2024-03-29T10:40:42.459 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.456+0000 7fd91affd640 10 client.?.objecter ms_handle_connect 0x7fd9240b7310 2024-03-29T10:40:42.459 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.456+0000 7fd91affd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:40:42.459 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.456+0000 7fd91affd640 1 -- 172.21.15.196:0/4014296949 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3912180328 0 0) 0x7fd9240b78c0 con 0x7fd92409ac40 2024-03-29T10:40:42.460 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.456+0000 7fd91affd640 10 monclient(hunting): _negotiate 2024-03-29T10:40:42.460 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.456+0000 7fd91affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:42.460 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.456+0000 7fd91affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:42.460 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.456+0000 7fd91affd640 10 monclient(hunting): my global_id is 4155 2024-03-29T10:40:42.460 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.456+0000 7fd91affd640 1 -- 172.21.15.196:0/4014296949 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd9240b7d50 con 0x7fd92409ac40 2024-03-29T10:40:42.460 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.456+0000 7fd91affd640 1 -- 172.21.15.196:0/4014296949 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3362160383 0 0) 0x7fd924146480 con 0x7fd9240b7310 2024-03-29T10:40:42.460 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.456+0000 7fd91affd640 10 monclient(hunting): _negotiate 2024-03-29T10:40:42.460 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.456+0000 7fd91affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:42.460 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.456+0000 7fd91affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:42.460 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.456+0000 7fd91affd640 10 monclient(hunting): my global_id is 4151 2024-03-29T10:40:42.460 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.456+0000 7fd91affd640 1 -- 172.21.15.196:0/4014296949 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd9240b78c0 con 0x7fd9240b7310 2024-03-29T10:40:42.460 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.457+0000 7fd91affd640 1 -- 172.21.15.196:0/4014296949 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 180722699 0 0) 0x7fd904004eb0 con 0x7fd9241d7de0 2024-03-29T10:40:42.460 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.457+0000 7fd91affd640 1 -- 172.21.15.196:0/4014296949 --> v1:172.21.15.196:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd924146480 con 0x7fd9241d7de0 2024-03-29T10:40:42.461 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.457+0000 7fd91affd640 1 -- 172.21.15.196:0/4014296949 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7fd91c001f40 con 0x7fd9241d7de0 2024-03-29T10:40:42.461 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.457+0000 7fd91affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:42.461 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.457+0000 7fd91affd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:40:42.461 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.457+0000 7fd91affd640 10 monclient(hunting): dump: 2024-03-29T10:40:42.461 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 1 2024-03-29T10:40:42.461 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:40:42.461 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:40:42.461 INFO:teuthology.orchestra.run.smithi196.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:40:42.461 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:42.461 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 1 2024-03-29T10:40:42.461 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:40:42.461 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:40:42.461 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:40:42.461 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-29T10:40:42.461 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.457+0000 7fd91affd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:40:42.462 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.457+0000 7fd91affd640 1 -- 172.21.15.196:0/4014296949 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2310225849 0 0) 0x7fd9240b7d50 con 0x7fd92409ac40 2024-03-29T10:40:42.462 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.457+0000 7fd91affd640 1 -- 172.21.15.196:0/4014296949 --> v1:172.21.15.88:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd904004eb0 con 0x7fd92409ac40 2024-03-29T10:40:42.462 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.457+0000 7fd91affd640 1 -- 172.21.15.196:0/4014296949 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7fd9140018b0 con 0x7fd92409ac40 2024-03-29T10:40:42.462 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.457+0000 7fd91affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:42.462 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.457+0000 7fd91affd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:40:42.462 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.457+0000 7fd91affd640 10 monclient(hunting): dump: 2024-03-29T10:40:42.462 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 1 2024-03-29T10:40:42.462 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:40:42.462 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:40:42.462 INFO:teuthology.orchestra.run.smithi196.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:40:42.462 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:42.463 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 1 2024-03-29T10:40:42.463 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:40:42.463 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:40:42.463 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:40:42.463 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-29T10:40:42.463 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.457+0000 7fd91affd640 1 -- 172.21.15.196:0/4014296949 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3763245505 0 0) 0x7fd9240b78c0 con 0x7fd9240b7310 2024-03-29T10:40:42.463 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.457+0000 7fd91affd640 1 -- 172.21.15.196:0/4014296949 --> v1:172.21.15.88:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd9240b7d50 con 0x7fd9240b7310 2024-03-29T10:40:42.463 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.457+0000 7fd91affd640 1 -- 172.21.15.196:0/4014296949 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7fd90c001860 con 0x7fd9240b7310 2024-03-29T10:40:42.463 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.457+0000 7fd91affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:42.464 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.457+0000 7fd91affd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:40:42.464 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.457+0000 7fd91affd640 10 monclient(hunting): dump: 2024-03-29T10:40:42.464 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 1 2024-03-29T10:40:42.464 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:40:42.464 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:40:42.464 INFO:teuthology.orchestra.run.smithi196.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:40:42.464 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:42.464 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 1 2024-03-29T10:40:42.464 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:40:42.464 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:40:42.464 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:40:42.464 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-29T10:40:42.464 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.457+0000 7fd91affd640 1 -- 172.21.15.196:0/4014296949 <== mon.1 v1:172.21.15.196:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 394923039 0 0) 0x7fd924146480 con 0x7fd9241d7de0 2024-03-29T10:40:42.465 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.457+0000 7fd91affd640 1 -- 172.21.15.196:0/4014296949 >> v1:172.21.15.88:6790/0 conn(0x7fd9240b7310 legacy=0x7fd9241d76d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:42.465 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.457+0000 7fd91affd640 1 -- 172.21.15.196:0/4014296949 >> v1:172.21.15.88:6789/0 conn(0x7fd92409ac40 legacy=0x7fd9240b6a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:42.465 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.457+0000 7fd91affd640 10 monclient: _finish_hunting 0 2024-03-29T10:40:42.465 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.457+0000 7fd91affd640 1 monclient: found mon.b 2024-03-29T10:40:42.465 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.457+0000 7fd91affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:40:42.465 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.457+0000 7fd91affd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T10:40:42.465 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.457+0000 7fd91affd640 1 -- 172.21.15.196:0/4014296949 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9241e68d0 con 0x7fd9241d7de0 2024-03-29T10:40:42.465 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.457+0000 7fd91affd640 10 monclient: _finish_auth 0 2024-03-29T10:40:42.465 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.457+0000 7fd91affd640 10 monclient: _check_auth_tickets 2024-03-29T10:40:42.465 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.457+0000 7fd91affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:40:42.465 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.457+0000 7fd928ae0640 5 monclient: authenticate success, global_id 4153 2024-03-29T10:40:42.465 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.457+0000 7fd928ae0640 10 monclient: _renew_subs 2024-03-29T10:40:42.465 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.457+0000 7fd928ae0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T10:40:42.465 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.457+0000 7fd928ae0640 1 -- 172.21.15.196:0/4014296949 --> v1:172.21.15.196:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9241e38a0 con 0x7fd9241d7de0 2024-03-29T10:40:42.466 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.457+0000 7fd928ae0640 10 client.4153.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:40:42.466 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.457+0000 7fd928ae0640 10 monclient: _renew_subs 2024-03-29T10:40:42.466 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.457+0000 7fd928ae0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T10:40:42.466 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.457+0000 7fd928ae0640 1 -- 172.21.15.196:0/4014296949 --> v1:172.21.15.196:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9241e3d80 con 0x7fd9241d7de0 2024-03-29T10:40:42.466 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.457+0000 7fd928ae0640 1 librados: init done 2024-03-29T10:40:42.466 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.457+0000 7fd91affd640 1 -- 172.21.15.196:0/4014296949 <== mon.1 v1:172.21.15.196:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd91c004e10 con 0x7fd9241d7de0 2024-03-29T10:40:42.466 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.457+0000 7fd91affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:40:42.466 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.457+0000 7fd91affd640 1 -- 172.21.15.196:0/4014296949 <== mon.1 v1:172.21.15.196:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7fd91c005120 con 0x7fd9241d7de0 2024-03-29T10:40:42.466 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.457+0000 7fd91affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:42.466 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.457+0000 7fd92159a640 4 set_mon_vals no callback set 2024-03-29T10:40:42.466 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.457+0000 7fd91affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:40:42.466 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.457+0000 7fd91affd640 10 monclient: dump: 2024-03-29T10:40:42.466 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 1 2024-03-29T10:40:42.466 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:40:42.467 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:40:42.467 INFO:teuthology.orchestra.run.smithi196.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:40:42.467 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:42.467 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 1 2024-03-29T10:40:42.467 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:40:42.467 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:40:42.467 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:40:42.467 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-29T10:40:42.467 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.457+0000 7fd91affd640 1 -- 172.21.15.196:0/4014296949 <== mon.1 v1:172.21.15.196:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fd91c005610 con 0x7fd9241d7de0 2024-03-29T10:40:42.467 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.457+0000 7fd91affd640 1 -- 172.21.15.196:0/4014296949 <== mon.1 v1:172.21.15.196:6789/0 8 ==== osd_map(9..9 src has 1..9) v4 ==== 1530+0+0 (unknown 3422872583 0 0) 0x7fd91c006db0 con 0x7fd9241d7de0 2024-03-29T10:40:42.467 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.457+0000 7fd91affd640 10 client.4153.objecter ms_dispatch 0x7fd924000b90 osd_map(9..9 src has 1..9) v4 2024-03-29T10:40:42.467 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.457+0000 7fd91affd640 3 client.4153.objecter handle_osd_map got epochs [9,9] > 0 2024-03-29T10:40:42.467 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.457+0000 7fd91affd640 3 client.4153.objecter handle_osd_map decoding full epoch 9 2024-03-29T10:40:42.467 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.457+0000 7fd91affd640 20 client.4153.objecter dump_active .. 0 homeless 2024-03-29T10:40:42.467 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.458+0000 7fd928ae0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:40:42.468 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.458+0000 7fd928ae0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:40:42.468 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.458+0000 7fd928ae0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T10:40:42.468 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.458+0000 7fd928ae0640 1 -- 172.21.15.196:0/4014296949 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd908005080 con 0x7fd9241d7de0 2024-03-29T10:40:42.468 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.461+0000 7fd91affd640 1 -- 172.21.15.196:0/4014296949 <== mon.1 v1:172.21.15.196:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fd91c006800 con 0x7fd9241d7de0 2024-03-29T10:40:42.468 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.461+0000 7fd91affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:40:42.468 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.461+0000 7fd91affd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:40:42.484 INFO:tasks.ceph.mgr.x.smithi196.stderr:2024-03-29T10:40:42.501+0000 7ff5dcc4b040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-29T10:40:42.518 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.534+0000 7fd928ae0640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "ce963d96-5a8e-44c3-99de-c0dbebe234ed", "id": 7}] 2024-03-29T10:40:42.518 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.534+0000 7fd928ae0640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "ce963d96-5a8e-44c3-99de-c0dbebe234ed", "id": 7}] 2024-03-29T10:40:42.518 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.534+0000 7fd928ae0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T10:40:42.518 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.534+0000 7fd928ae0640 1 -- 172.21.15.196:0/4014296949 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "osd new", "uuid": "ce963d96-5a8e-44c3-99de-c0dbebe234ed", "id": 7} v 0) v1 -- 0x7fd9080058b0 con 0x7fd9241d7de0 2024-03-29T10:40:42.522 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.538+0000 7fd91affd640 1 -- 172.21.15.196:0/4014296949 <== mon.1 v1:172.21.15.196:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "ce963d96-5a8e-44c3-99de-c0dbebe234ed", "id": 7}]=0 v10) v1 ==== 112+0+2 (unknown 3221964219 0 1371665406) 0x7fd91c006a70 con 0x7fd9241d7de0 2024-03-29T10:40:42.522 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.538+0000 7fd91affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "ce963d96-5a8e-44c3-99de-c0dbebe234ed", "id": 7}] 2024-03-29T10:40:42.522 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.538+0000 7fd91affd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:40:42.523 INFO:teuthology.orchestra.run.smithi196.stdout:7 2024-03-29T10:40:42.524 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.539+0000 7fd928ae0640 10 librados: watch_flush enter 2024-03-29T10:40:42.524 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.539+0000 7fd928ae0640 10 librados: watch_flush exit 2024-03-29T10:40:42.524 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.539+0000 7fd928ae0640 20 client.4153.objecter shutdown clearing up homeless session... 2024-03-29T10:40:42.524 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.539+0000 7fd928ae0640 10 client.4153.objecter successfully canceled tick 2024-03-29T10:40:42.524 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.539+0000 7fd928ae0640 10 monclient: shutdown 2024-03-29T10:40:42.524 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.539+0000 7fd928ae0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:40:42.524 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.539+0000 7fd928ae0640 1 -- 172.21.15.196:0/4014296949 >> v1:172.21.15.196:6789/0 conn(0x7fd9241d7de0 legacy=0x7fd9241e2180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:42.524 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.539+0000 7fd928ae0640 1 -- 172.21.15.196:0/4014296949 shutdown_connections 2024-03-29T10:40:42.524 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.539+0000 7fd928ae0640 1 -- 172.21.15.196:0/4014296949 >> 172.21.15.196:0/4014296949 conn(0x7fd924066b00 msgr2=0x7fd92413a3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:40:42.524 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.539+0000 7fd928ae0640 1 -- 172.21.15.196:0/4014296949 shutdown_connections 2024-03-29T10:40:42.524 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.539+0000 7fd928ae0640 1 -- 172.21.15.196:0/4014296949 wait complete. 2024-03-29T10:40:42.525 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T10:40:42.539+0000 7fd928ae0640 1 librados: shutdown 2024-03-29T10:40:42.533 INFO:tasks.ceph.osd.0:Restarting daemon 2024-03-29T10:40:42.534 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-03-29T10:40:42.539 INFO:tasks.ceph.osd.0:Started 2024-03-29T10:40:42.540 INFO:tasks.ceph.osd.1:Restarting daemon 2024-03-29T10:40:42.540 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-03-29T10:40:42.542 INFO:tasks.ceph.osd.1:Started 2024-03-29T10:40:42.542 INFO:tasks.ceph.osd.2:Restarting daemon 2024-03-29T10:40:42.542 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-03-29T10:40:42.544 INFO:tasks.ceph.osd.2:Started 2024-03-29T10:40:42.544 INFO:tasks.ceph.osd.3:Restarting daemon 2024-03-29T10:40:42.544 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-03-29T10:40:42.546 INFO:tasks.ceph.osd.3:Started 2024-03-29T10:40:42.546 INFO:tasks.ceph.osd.4:Restarting daemon 2024-03-29T10:40:42.546 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-03-29T10:40:42.547 INFO:tasks.ceph.mgr.y.smithi088.stderr:2024-03-29T10:40:42.564+0000 7fca56440040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-29T10:40:42.576 INFO:tasks.ceph.osd.4:Started 2024-03-29T10:40:42.576 INFO:tasks.ceph.osd.5:Restarting daemon 2024-03-29T10:40:42.576 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-03-29T10:40:42.578 INFO:tasks.ceph.osd.5:Started 2024-03-29T10:40:42.578 INFO:tasks.ceph.osd.6:Restarting daemon 2024-03-29T10:40:42.578 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2024-03-29T10:40:42.580 INFO:tasks.ceph.osd.6:Started 2024-03-29T10:40:42.580 INFO:tasks.ceph.osd.7:Restarting daemon 2024-03-29T10:40:42.581 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2024-03-29T10:40:42.582 INFO:tasks.ceph.osd.7:Started 2024-03-29T10:40:42.583 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T10:40:42.620 INFO:tasks.ceph.mgr.y.smithi088.stderr:2024-03-29T10:40:42.636+0000 7fca56440040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-29T10:40:42.631 INFO:tasks.ceph.mgr.x.smithi196.stderr:2024-03-29T10:40:42.647+0000 7ff5dcc4b040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-29T10:40:42.685 INFO:tasks.ceph.mgr.x.smithi196.stderr:2024-03-29T10:40:42.701+0000 7ff5dcc4b040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-29T10:40:42.694 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:42.711+0000 7f4788e93640 10 monclient: get_monmap_and_config 2024-03-29T10:40:42.694 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:42.711+0000 7f4788e93640 10 monclient: build_initial_monmap 2024-03-29T10:40:42.695 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:42.711+0000 7f4788e93640 10 monclient: monmap: 2024-03-29T10:40:42.695 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2024-03-29T10:40:42.695 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:40:42.695 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-03-29T10:40:42.712231+0000 2024-03-29T10:40:42.695 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-03-29T10:40:42.712231+0000 2024-03-29T10:40:42.695 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2024-03-29T10:40:42.695 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2024-03-29T10:40:42.695 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-03-29T10:40:42.695 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-29T10:40:42.695 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-b 2024-03-29T10:40:42.695 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-03-29T10:40:42.704 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:42.711+0000 7f4788e93640 1 Processor -- start 2024-03-29T10:40:42.704 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:42.711+0000 7f4788e93640 1 -- start start 2024-03-29T10:40:42.704 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:42.711+0000 7f4788e93640 10 monclient: init 2024-03-29T10:40:42.704 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:42.712+0000 7f4788e93640 10 monclient: _reopen_session rank -1 2024-03-29T10:40:42.704 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:42.712+0000 7f4788e93640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:40:42.704 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:42.712+0000 7f4788e93640 10 monclient(hunting): picked mon.noname-a con 0x7f478409ac40 addr v1:172.21.15.88:6789/0 2024-03-29T10:40:42.704 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:42.712+0000 7f4788e93640 10 monclient(hunting): picked mon.noname-c con 0x7f478409c870 addr v1:172.21.15.196:6789/0 2024-03-29T10:40:42.705 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:42.712+0000 7f4788e93640 10 monclient(hunting): picked mon.noname-b con 0x7f47840b4360 addr v1:172.21.15.88:6790/0 2024-03-29T10:40:42.705 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:42.712+0000 7f4788e93640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f478409dec0 con 0x7f478409ac40 2024-03-29T10:40:42.705 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:42.712+0000 7f4788e93640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f478409da30 con 0x7f478409c870 2024-03-29T10:40:42.705 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:42.712+0000 7f4788e93640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47840b39c0 con 0x7f47840b4360 2024-03-29T10:40:42.705 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:42.712+0000 7f4788e93640 10 monclient(hunting): _renew_subs 2024-03-29T10:40:42.705 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:42.712+0000 7f4788e93640 10 monclient(hunting): authenticate will time out at 764.948413s 2024-03-29T10:40:42.706 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:42.712+0000 7f478259c640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f478409ac40 0x7f47840b4a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:58734/0 (socket says 172.21.15.88:58734) 2024-03-29T10:40:42.706 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:42.712+0000 7f478259c640 1 -- 172.21.15.88:0/1395667000 learned_addr learned my addr 172.21.15.88:0/1395667000 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-29T10:40:42.706 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:42.712+0000 7f4781d9b640 1 -- 172.21.15.88:0/1395667000 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 92970840 0 0) 0x7f4774001110 con 0x7f478409ac40 2024-03-29T10:40:42.706 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:42.712+0000 7f4781d9b640 10 monclient(hunting): _negotiate 2024-03-29T10:40:42.706 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:42.712+0000 7f4781d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:42.706 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:42.712+0000 7f4781d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:42.706 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:42.712+0000 7f4781d9b640 10 monclient(hunting): my global_id is 4182 2024-03-29T10:40:42.706 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:42.712+0000 7f4781d9b640 1 -- 172.21.15.88:0/1395667000 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4770003520 con 0x7f478409ac40 2024-03-29T10:40:42.706 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:42.712+0000 7f4781d9b640 1 -- 172.21.15.88:0/1395667000 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2423860166 0 0) 0x7f47840b39c0 con 0x7f47840b4360 2024-03-29T10:40:42.706 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:42.712+0000 7f4781d9b640 10 monclient(hunting): _negotiate 2024-03-29T10:40:42.706 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:42.712+0000 7f4781d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:42.707 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:42.712+0000 7f4781d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:42.707 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:42.712+0000 7f4781d9b640 10 monclient(hunting): my global_id is 4178 2024-03-29T10:40:42.707 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:42.712+0000 7f4781d9b640 1 -- 172.21.15.88:0/1395667000 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4774001110 con 0x7f47840b4360 2024-03-29T10:40:42.707 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:42.713+0000 7f4781d9b640 1 -- 172.21.15.88:0/1395667000 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1582760949 0 0) 0x7f478409da30 con 0x7f478409c870 2024-03-29T10:40:42.707 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:42.713+0000 7f4781d9b640 10 monclient(hunting): _negotiate 2024-03-29T10:40:42.707 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:42.713+0000 7f4781d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:42.707 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:42.713+0000 7f4781d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:42.707 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:42.713+0000 7f4781d9b640 10 monclient(hunting): my global_id is 4177 2024-03-29T10:40:42.707 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:42.713+0000 7f4781d9b640 1 -- 172.21.15.88:0/1395667000 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f47840b39c0 con 0x7f478409c870 2024-03-29T10:40:42.707 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:42.713+0000 7f4781d9b640 1 -- 172.21.15.88:0/1395667000 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 312536138 0 0) 0x7f4774001110 con 0x7f47840b4360 2024-03-29T10:40:42.707 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:42.713+0000 7f4781d9b640 1 -- 172.21.15.88:0/1395667000 >> v1:172.21.15.196:6789/0 conn(0x7f478409c870 legacy=0x7f478413f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:42.707 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:42.713+0000 7f4781d9b640 1 -- 172.21.15.88:0/1395667000 >> v1:172.21.15.88:6789/0 conn(0x7f478409ac40 legacy=0x7f47840b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:42.707 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:42.713+0000 7f4781d9b640 10 monclient: _finish_hunting 0 2024-03-29T10:40:42.707 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:42.713+0000 7f4781d9b640 1 monclient: found mon.noname-b 2024-03-29T10:40:42.708 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:42.713+0000 7f4781d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:40:42.708 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:42.713+0000 7f4781d9b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.88:6790/0 2024-03-29T10:40:42.708 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:42.713+0000 7f4781d9b640 1 -- 172.21.15.88:0/1395667000 --> v1:172.21.15.88:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47840b5160 con 0x7f47840b4360 2024-03-29T10:40:42.708 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:42.713+0000 7f4781d9b640 10 monclient: _finish_auth 0 2024-03-29T10:40:42.708 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:42.713+0000 7f4781d9b640 10 monclient: _check_auth_tickets 2024-03-29T10:40:42.708 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:42.713+0000 7f4781d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:40:42.708 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:42.713+0000 7f4788e93640 5 monclient: authenticate success, global_id 4178 2024-03-29T10:40:42.708 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:42.713+0000 7f4781d9b640 1 -- 172.21.15.88:0/1395667000 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f477c001920 con 0x7f47840b4360 2024-03-29T10:40:42.708 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:42.713+0000 7f4788e93640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:40:42.708 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:42.713+0000 7f4781d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:42.708 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:42.713+0000 7f4781d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:40:42.708 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:42.713+0000 7f4781d9b640 10 monclient: dump: 2024-03-29T10:40:42.708 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-03-29T10:40:42.708 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:40:42.708 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:40:42.709 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:40:42.709 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:42.709 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2024-03-29T10:40:42.709 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:40:42.709 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:40:42.709 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:40:42.709 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-03-29T10:40:42.709 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:42.713+0000 7f4788e93640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:40:42.709 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:42.713+0000 7f4781d9b640 1 -- 172.21.15.88:0/1395667000 <== mon.2 v1:172.21.15.88:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f477c001b00 con 0x7f47840b4360 2024-03-29T10:40:42.709 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:42.713+0000 7f4781d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:40:42.709 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:42.713+0000 7f4788e93640 10 monclient: get_monmap_and_config success 2024-03-29T10:40:42.709 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:42.713+0000 7f4781d9b640 1 -- 172.21.15.88:0/1395667000 <== mon.2 v1:172.21.15.88:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f477c001f50 con 0x7f47840b4360 2024-03-29T10:40:42.709 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:42.713+0000 7f4788e93640 4 set_mon_vals no callback set 2024-03-29T10:40:42.709 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:42.713+0000 7f4781d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:42.710 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:42.713+0000 7f4781d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:40:42.710 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:42.713+0000 7f4781d9b640 10 monclient: dump: 2024-03-29T10:40:42.710 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-03-29T10:40:42.710 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:40:42.710 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:40:42.710 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:40:42.710 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:42.710 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2024-03-29T10:40:42.710 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:40:42.710 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:40:42.710 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:40:42.710 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-03-29T10:40:42.710 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:42.713+0000 7f4788e93640 10 monclient: shutdown 2024-03-29T10:40:42.710 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:42.713+0000 7f4788e93640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:40:42.711 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:42.713+0000 7f4788e93640 1 -- 172.21.15.88:0/1395667000 >> v1:172.21.15.88:6790/0 conn(0x7f47840b4360 legacy=0x7f478414d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:42.711 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:42.713+0000 7f4788e93640 1 -- 172.21.15.88:0/1395667000 shutdown_connections 2024-03-29T10:40:42.711 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:42.713+0000 7f4788e93640 1 -- 172.21.15.88:0/1395667000 >> 172.21.15.88:0/1395667000 conn(0x7f4784066b00 msgr2=0x7f4784066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:40:42.711 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:42.714+0000 7f4788e93640 1 -- 172.21.15.88:0/1395667000 shutdown_connections 2024-03-29T10:40:42.711 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:42.714+0000 7f4788e93640 1 -- 172.21.15.88:0/1395667000 wait complete. 2024-03-29T10:40:42.711 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:42.714+0000 7f4788e93640 10 monclient: build_initial_monmap 2024-03-29T10:40:42.711 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:42.714+0000 7f4788e93640 10 monclient: monmap: 2024-03-29T10:40:42.711 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2024-03-29T10:40:42.711 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:40:42.711 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 0.000000 2024-03-29T10:40:42.711 INFO:teuthology.orchestra.run.smithi088.stderr:created 0.000000 2024-03-29T10:40:42.711 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2024-03-29T10:40:42.711 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2024-03-29T10:40:42.712 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-03-29T10:40:42.712 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-29T10:40:42.712 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-c 2024-03-29T10:40:42.712 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-03-29T10:40:42.712 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:42.715+0000 7f4788e93640 1 librados: starting msgr at 2024-03-29T10:40:42.712 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:42.715+0000 7f4788e93640 1 librados: starting objecter 2024-03-29T10:40:42.712 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:42.715+0000 7f4788e93640 1 Processor -- start 2024-03-29T10:40:42.712 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:42.715+0000 7f4788e93640 1 -- start start 2024-03-29T10:40:42.712 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:42.715+0000 7f4788e93640 1 librados: setting wanted keys 2024-03-29T10:40:42.712 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:42.715+0000 7f4788e93640 1 librados: calling monclient init 2024-03-29T10:40:42.712 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:42.715+0000 7f4788e93640 10 monclient: init 2024-03-29T10:40:42.712 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:42.715+0000 7f4788e93640 10 monclient: _reopen_session rank -1 2024-03-29T10:40:42.712 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:42.715+0000 7f4788e93640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:40:42.712 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:42.715+0000 7f4788e93640 10 monclient(hunting): picked mon.noname-a con 0x7f478409ac40 addr v1:172.21.15.88:6789/0 2024-03-29T10:40:42.712 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:42.715+0000 7f4788e93640 10 monclient(hunting): picked mon.noname-b con 0x7f47840b08d0 addr v1:172.21.15.196:6789/0 2024-03-29T10:40:42.713 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:42.715+0000 7f4788e93640 10 monclient(hunting): picked mon.noname-c con 0x7f47841d7c00 addr v1:172.21.15.88:6790/0 2024-03-29T10:40:42.713 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:42.715+0000 7f4788e93640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4784140da0 con 0x7f478409ac40 2024-03-29T10:40:42.713 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:42.715+0000 7f4788e93640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4784141010 con 0x7f47840b08d0 2024-03-29T10:40:42.713 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:42.715+0000 7f4788e93640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47841d6fd0 con 0x7f47841d7c00 2024-03-29T10:40:42.713 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:42.715+0000 7f4788e93640 10 monclient(hunting): _renew_subs 2024-03-29T10:40:42.713 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:42.715+0000 7f4788e93640 10 monclient(hunting): authenticate will time out at 764.951641s 2024-03-29T10:40:42.713 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:42.715+0000 7f4782d9d640 1 --1- >> v1:172.21.15.88:6790/0 conn(0x7f47841d7c00 0x7f4784136d60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6790/0 says I am v1:172.21.15.88:35642/0 (socket says 172.21.15.88:35642) 2024-03-29T10:40:42.713 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:42.715+0000 7f4782d9d640 1 -- 172.21.15.88:0/1504880939 learned_addr learned my addr 172.21.15.88:0/1504880939 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-29T10:40:42.713 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:42.715+0000 7f477affd640 10 client.?.objecter ms_handle_connect 0x7f47841d7c00 2024-03-29T10:40:42.713 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:42.715+0000 7f477affd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:40:42.713 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:42.715+0000 7f477affd640 10 client.?.objecter ms_handle_connect 0x7f478409ac40 2024-03-29T10:40:42.713 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:42.715+0000 7f477affd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:40:42.713 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:42.716+0000 7f477affd640 10 client.?.objecter ms_handle_connect 0x7f47840b08d0 2024-03-29T10:40:42.714 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:42.716+0000 7f477affd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:40:42.714 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:42.716+0000 7f477affd640 1 -- 172.21.15.88:0/1504880939 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1490460645 0 0) 0x7f47841d6fd0 con 0x7f47841d7c00 2024-03-29T10:40:42.714 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:42.716+0000 7f477affd640 10 monclient(hunting): _negotiate 2024-03-29T10:40:42.714 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:42.716+0000 7f477affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:42.714 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:42.716+0000 7f477affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:42.714 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:42.716+0000 7f477affd640 10 monclient(hunting): my global_id is 4181 2024-03-29T10:40:42.714 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:42.716+0000 7f477affd640 1 -- 172.21.15.88:0/1504880939 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4764004eb0 con 0x7f47841d7c00 2024-03-29T10:40:42.714 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:42.716+0000 7f477affd640 1 -- 172.21.15.88:0/1504880939 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1167625895 0 0) 0x7f4784140da0 con 0x7f478409ac40 2024-03-29T10:40:42.714 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:42.716+0000 7f477affd640 10 monclient(hunting): _negotiate 2024-03-29T10:40:42.714 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:42.716+0000 7f477affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:42.714 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:42.716+0000 7f477affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:42.714 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:42.716+0000 7f477affd640 10 monclient(hunting): my global_id is 4185 2024-03-29T10:40:42.714 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:42.716+0000 7f477affd640 1 -- 172.21.15.88:0/1504880939 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f47841d6fd0 con 0x7f478409ac40 2024-03-29T10:40:42.714 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:42.716+0000 7f477affd640 1 -- 172.21.15.88:0/1504880939 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4240598094 0 0) 0x7f4784141010 con 0x7f47840b08d0 2024-03-29T10:40:42.715 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:42.716+0000 7f477affd640 10 monclient(hunting): _negotiate 2024-03-29T10:40:42.715 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:42.716+0000 7f477affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:42.715 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:42.716+0000 7f477affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:42.715 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:42.716+0000 7f477affd640 10 monclient(hunting): my global_id is 4180 2024-03-29T10:40:42.715 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:42.716+0000 7f477affd640 1 -- 172.21.15.88:0/1504880939 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4784140da0 con 0x7f47840b08d0 2024-03-29T10:40:42.715 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:42.716+0000 7f477affd640 1 -- 172.21.15.88:0/1504880939 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1283634087 0 0) 0x7f4764004eb0 con 0x7f47841d7c00 2024-03-29T10:40:42.715 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:42.716+0000 7f477affd640 1 -- 172.21.15.88:0/1504880939 --> v1:172.21.15.88:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4784141010 con 0x7f47841d7c00 2024-03-29T10:40:42.715 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:42.716+0000 7f477affd640 1 -- 172.21.15.88:0/1504880939 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1769930592 0 0) 0x7f47841d6fd0 con 0x7f478409ac40 2024-03-29T10:40:42.715 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:42.716+0000 7f477affd640 1 -- 172.21.15.88:0/1504880939 --> v1:172.21.15.88:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4764004eb0 con 0x7f478409ac40 2024-03-29T10:40:42.715 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:42.716+0000 7f477affd640 1 -- 172.21.15.88:0/1504880939 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f477c001f50 con 0x7f47841d7c00 2024-03-29T10:40:42.715 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:42.716+0000 7f477affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:42.715 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:42.716+0000 7f477affd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:40:42.716 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:42.716+0000 7f477affd640 10 monclient(hunting): dump: 2024-03-29T10:40:42.716 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-03-29T10:40:42.716 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:40:42.716 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:40:42.716 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:40:42.716 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:42.716 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2024-03-29T10:40:42.716 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:40:42.716 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:40:42.716 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:40:42.716 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-03-29T10:40:42.716 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:42.716+0000 7f477affd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:40:42.716 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:42.716+0000 7f477affd640 1 -- 172.21.15.88:0/1504880939 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f4774001150 con 0x7f478409ac40 2024-03-29T10:40:42.716 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:42.716+0000 7f477affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:42.716 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:42.716+0000 7f477affd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:40:42.717 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:42.716+0000 7f477affd640 10 monclient(hunting): dump: 2024-03-29T10:40:42.717 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-03-29T10:40:42.717 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:40:42.717 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:40:42.717 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:40:42.717 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:42.717 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2024-03-29T10:40:42.717 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:40:42.717 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:40:42.717 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:40:42.717 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-03-29T10:40:42.717 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:42.716+0000 7f477affd640 1 -- 172.21.15.88:0/1504880939 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2694349497 0 0) 0x7f4784140da0 con 0x7f47840b08d0 2024-03-29T10:40:42.717 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:42.716+0000 7f477affd640 1 -- 172.21.15.88:0/1504880939 --> v1:172.21.15.196:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f47841d6fd0 con 0x7f47840b08d0 2024-03-29T10:40:42.717 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:42.716+0000 7f477affd640 1 -- 172.21.15.88:0/1504880939 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f476c001830 con 0x7f47840b08d0 2024-03-29T10:40:42.717 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:42.716+0000 7f477affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:42.718 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:42.716+0000 7f477affd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T10:40:42.718 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:42.716+0000 7f477affd640 10 monclient(hunting): dump: 2024-03-29T10:40:42.718 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-03-29T10:40:42.718 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:40:42.718 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:40:42.718 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:40:42.718 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:42.718 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2024-03-29T10:40:42.718 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:40:42.718 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:40:42.718 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:40:42.718 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-03-29T10:40:42.718 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:42.716+0000 7f477affd640 1 -- 172.21.15.88:0/1504880939 <== mon.2 v1:172.21.15.88:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2479281802 0 0) 0x7f4784141010 con 0x7f47841d7c00 2024-03-29T10:40:42.718 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:42.716+0000 7f477affd640 1 -- 172.21.15.88:0/1504880939 >> v1:172.21.15.196:6789/0 conn(0x7f47840b08d0 legacy=0x7f47841d74f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:42.719 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:42.716+0000 7f477affd640 1 -- 172.21.15.88:0/1504880939 >> v1:172.21.15.88:6789/0 conn(0x7f478409ac40 legacy=0x7f47840b1350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:42.719 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:42.716+0000 7f477affd640 10 monclient: _finish_hunting 0 2024-03-29T10:40:42.719 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:42.716+0000 7f477affd640 1 monclient: found mon.c 2024-03-29T10:40:42.719 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:42.716+0000 7f477affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:40:42.719 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:42.716+0000 7f477affd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-03-29T10:40:42.719 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:42.716+0000 7f477affd640 1 -- 172.21.15.88:0/1504880939 --> v1:172.21.15.88:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47840add50 con 0x7f47841d7c00 2024-03-29T10:40:42.719 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:42.716+0000 7f477affd640 10 monclient: _finish_auth 0 2024-03-29T10:40:42.719 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:42.716+0000 7f477affd640 10 monclient: _check_auth_tickets 2024-03-29T10:40:42.719 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:42.716+0000 7f477affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:40:42.719 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:42.716+0000 7f4788e93640 5 monclient: authenticate success, global_id 4181 2024-03-29T10:40:42.719 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:42.716+0000 7f4788e93640 10 monclient: _renew_subs 2024-03-29T10:40:42.719 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:42.716+0000 7f4788e93640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-03-29T10:40:42.719 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:42.716+0000 7f4788e93640 1 -- 172.21.15.88:0/1504880939 --> v1:172.21.15.88:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4784137470 con 0x7f47841d7c00 2024-03-29T10:40:42.719 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:42.716+0000 7f4788e93640 10 client.4181.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:40:42.719 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:42.716+0000 7f4788e93640 10 monclient: _renew_subs 2024-03-29T10:40:42.720 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:42.716+0000 7f4788e93640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-03-29T10:40:42.720 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:42.716+0000 7f4788e93640 1 -- 172.21.15.88:0/1504880939 --> v1:172.21.15.88:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f47841e3900 con 0x7f47841d7c00 2024-03-29T10:40:42.720 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:42.716+0000 7f4788e93640 1 librados: init done 2024-03-29T10:40:42.720 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:42.716+0000 7f477affd640 1 -- 172.21.15.88:0/1504880939 <== mon.2 v1:172.21.15.88:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f477c003c20 con 0x7f47841d7c00 2024-03-29T10:40:42.720 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:42.716+0000 7f477affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:40:42.720 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:42.716+0000 7f477affd640 1 -- 172.21.15.88:0/1504880939 <== mon.2 v1:172.21.15.88:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f477c0020b0 con 0x7f47841d7c00 2024-03-29T10:40:42.720 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:42.716+0000 7f477affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:42.720 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:42.716+0000 7f477affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:40:42.720 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:42.716+0000 7f477affd640 10 monclient: dump: 2024-03-29T10:40:42.720 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-03-29T10:40:42.720 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:40:42.720 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:40:42.720 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:40:42.720 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:42.720 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2024-03-29T10:40:42.721 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:40:42.721 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:40:42.721 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:40:42.721 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-03-29T10:40:42.721 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:42.716+0000 7f478159a640 4 set_mon_vals no callback set 2024-03-29T10:40:42.721 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:42.716+0000 7f477affd640 1 -- 172.21.15.88:0/1504880939 <== mon.2 v1:172.21.15.88:6790/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f477c004440 con 0x7f47841d7c00 2024-03-29T10:40:42.721 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:42.716+0000 7f477affd640 1 -- 172.21.15.88:0/1504880939 <== mon.2 v1:172.21.15.88:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4257673582 0 0) 0x7f477c0061d0 con 0x7f47841d7c00 2024-03-29T10:40:42.721 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:42.717+0000 7f477affd640 10 client.4181.objecter ms_dispatch 0x7f4784000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:40:42.721 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:42.717+0000 7f477affd640 3 client.4181.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:40:42.721 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:42.717+0000 7f477affd640 3 client.4181.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:40:42.721 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:42.717+0000 7f477affd640 20 client.4181.objecter dump_active .. 0 homeless 2024-03-29T10:40:42.721 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:42.718+0000 7f4788e93640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:40:42.721 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:42.718+0000 7f4788e93640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:40:42.721 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:42.718+0000 7f4788e93640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-03-29T10:40:42.722 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:42.718+0000 7f4788e93640 1 -- 172.21.15.88:0/1504880939 --> v1:172.21.15.88:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4768005080 con 0x7f47841d7c00 2024-03-29T10:40:42.722 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:42.720+0000 7f477affd640 1 -- 172.21.15.88:0/1504880939 <== mon.2 v1:172.21.15.88:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f477c0046f0 con 0x7f47841d7c00 2024-03-29T10:40:42.722 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:42.720+0000 7f477affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:40:42.722 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:42.720+0000 7f477affd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:40:42.755 INFO:tasks.ceph.mgr.y.smithi088.stderr:2024-03-29T10:40:42.771+0000 7fca56440040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-29T10:40:42.777 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:42.793+0000 7f4788e93640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:40:42.777 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:42.793+0000 7f4788e93640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:40:42.777 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:42.793+0000 7f4788e93640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-03-29T10:40:42.777 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:42.793+0000 7f4788e93640 1 -- 172.21.15.88:0/1504880939 --> v1:172.21.15.88:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4768005330 con 0x7f47841d7c00 2024-03-29T10:40:42.777 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:42.793+0000 7f477affd640 1 -- 172.21.15.88:0/1504880939 <== mon.2 v1:172.21.15.88:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 344506374) 0x7f477c005c10 con 0x7f47841d7c00 2024-03-29T10:40:42.777 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:42.793+0000 7f477affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:40:42.777 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:42.793+0000 7f477affd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:40:42.778 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-03-29T10:40:42.778 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":10,"fsid":"869f5188-f92b-42dd-a46d-8c646e0388a9","created":"2024-03-29T10:40:40.285087+0000","modified":"2024-03-29T10:40:42.536230+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:40:42.536230+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a26c0c1c-8de0-438c-a3f6-3df10ea61574","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5de24915-8dec-49b6-8b16-f3170197c591","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ac054af3-d589-4565-9644-e6af75975c53","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3ec4f045-c714-4a22-bd1a-e07330b45ab3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"cddb9867-8e30-4951-9b13-3d1e78b9c45a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dc6d9911-785d-432f-8b53-56a1ccab302f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"77c88437-b587-496e-8b4d-1824ea8b5305","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ce963d96-5a8e-44c3-99de-c0dbebe234ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:40:42.779 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:42.794+0000 7f4788e93640 10 librados: watch_flush enter 2024-03-29T10:40:42.779 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:42.794+0000 7f4788e93640 10 librados: watch_flush exit 2024-03-29T10:40:42.779 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:42.794+0000 7f4788e93640 20 client.4181.objecter shutdown clearing up homeless session... 2024-03-29T10:40:42.779 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:42.794+0000 7f4788e93640 10 client.4181.objecter successfully canceled tick 2024-03-29T10:40:42.779 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:42.794+0000 7f4788e93640 10 monclient: shutdown 2024-03-29T10:40:42.779 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:42.794+0000 7f4788e93640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:40:42.780 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:42.794+0000 7f4788e93640 1 -- 172.21.15.88:0/1504880939 >> v1:172.21.15.88:6790/0 conn(0x7f47841d7c00 legacy=0x7f4784136d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:42.780 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:42.794+0000 7f4788e93640 1 -- 172.21.15.88:0/1504880939 shutdown_connections 2024-03-29T10:40:42.780 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:42.794+0000 7f4788e93640 1 -- 172.21.15.88:0/1504880939 >> 172.21.15.88:0/1504880939 conn(0x7f4784066b00 msgr2=0x7f478413d7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:40:42.780 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:42.794+0000 7f4788e93640 1 -- 172.21.15.88:0/1504880939 shutdown_connections 2024-03-29T10:40:42.780 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:42.794+0000 7f4788e93640 1 -- 172.21.15.88:0/1504880939 wait complete. 2024-03-29T10:40:42.780 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-29T10:40:42.794+0000 7f4788e93640 1 librados: shutdown 2024-03-29T10:40:42.789 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-03-29T10:40:42.789 INFO:tasks.ceph:Waiting for OSDs to come up 2024-03-29T10:40:42.796 INFO:tasks.ceph.mgr.x.smithi196.stderr:2024-03-29T10:40:42.812+0000 7ff5dcc4b040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-29T10:40:42.853 INFO:tasks.ceph.mgr.y.smithi088.stderr:2024-03-29T10:40:42.869+0000 7fca56440040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-29T10:40:42.890 INFO:tasks.ceph.mgr.x.smithi196.stderr:2024-03-29T10:40:42.907+0000 7ff5dcc4b040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-29T10:40:43.008 INFO:tasks.ceph.mgr.y.smithi088.stderr:2024-03-29T10:40:43.024+0000 7fca56440040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-29T10:40:43.047 INFO:tasks.ceph.mgr.x.smithi196.stderr:2024-03-29T10:40:43.063+0000 7ff5dcc4b040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-29T10:40:43.049 INFO:tasks.ceph.mgr.y.smithi088.stderr:2024-03-29T10:40:43.066+0000 7fca56440040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-29T10:40:43.088 INFO:tasks.ceph.mgr.x.smithi196.stderr:2024-03-29T10:40:43.104+0000 7ff5dcc4b040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-29T10:40:43.150 INFO:tasks.ceph.osd.3.smithi088.stderr:2024-03-29T10:40:43.166+0000 7f5366413640 -1 Falling back to public interface 2024-03-29T10:40:43.153 INFO:tasks.ceph.osd.0.smithi088.stderr:2024-03-29T10:40:43.169+0000 7f5811213640 -1 Falling back to public interface 2024-03-29T10:40:43.157 INFO:tasks.ceph.osd.1.smithi088.stderr:2024-03-29T10:40:43.173+0000 7fa68661c640 -1 Falling back to public interface 2024-03-29T10:40:43.161 INFO:tasks.ceph.osd.2.smithi088.stderr:2024-03-29T10:40:43.177+0000 7f2263a0e640 -1 Falling back to public interface 2024-03-29T10:40:43.192 INFO:tasks.ceph.osd.6.smithi196.stderr:2024-03-29T10:40:43.208+0000 7f055be68640 -1 Falling back to public interface 2024-03-29T10:40:43.195 INFO:tasks.ceph.osd.4.smithi196.stderr:2024-03-29T10:40:43.212+0000 7f73220f9640 -1 Falling back to public interface 2024-03-29T10:40:43.200 INFO:tasks.ceph.osd.7.smithi196.stderr:2024-03-29T10:40:43.216+0000 7fdda0a15640 -1 Falling back to public interface 2024-03-29T10:40:43.203 INFO:tasks.ceph.osd.5.smithi196.stderr:2024-03-29T10:40:43.219+0000 7fc4a100c640 -1 Falling back to public interface 2024-03-29T10:40:43.209 INFO:tasks.ceph.mgr.y.smithi088.stderr:2024-03-29T10:40:43.225+0000 7fca56440040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-29T10:40:43.243 INFO:tasks.ceph.mgr.x.smithi196.stderr:2024-03-29T10:40:43.259+0000 7ff5dcc4b040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-29T10:40:43.595 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:40:43.685 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:43.701+0000 7f719c379640 10 monclient: get_monmap_and_config 2024-03-29T10:40:43.685 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:43.701+0000 7f719c379640 10 monclient: build_initial_monmap 2024-03-29T10:40:43.685 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:43.701+0000 7f719c379640 10 monclient: monmap: 2024-03-29T10:40:43.685 INFO:teuthology.misc.health.smithi088.stderr:epoch 0 2024-03-29T10:40:43.685 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:40:43.685 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:43.702348+0000 2024-03-29T10:40:43.685 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:43.702348+0000 2024-03-29T10:40:43.686 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 0 (unknown) 2024-03-29T10:40:43.686 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:40:43.686 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-03-29T10:40:43.686 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-29T10:40:43.686 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-b 2024-03-29T10:40:43.686 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:40:43.686 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:43.701+0000 7f719c379640 1 Processor -- start 2024-03-29T10:40:43.686 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:43.701+0000 7f719c379640 1 -- start start 2024-03-29T10:40:43.686 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:43.701+0000 7f719c379640 10 monclient: init 2024-03-29T10:40:43.693 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:43.702+0000 7f719c379640 10 monclient: _reopen_session rank -1 2024-03-29T10:40:43.693 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:43.702+0000 7f719c379640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:40:43.694 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:43.702+0000 7f719c379640 10 monclient(hunting): picked mon.noname-c con 0x7f71941457f0 addr v1:172.21.15.196:6789/0 2024-03-29T10:40:43.694 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:43.702+0000 7f719c379640 10 monclient(hunting): picked mon.noname-b con 0x7f719413f650 addr v1:172.21.15.88:6790/0 2024-03-29T10:40:43.694 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:43.702+0000 7f719c379640 10 monclient(hunting): picked mon.noname-a con 0x7f71940a9d70 addr v1:172.21.15.88:6789/0 2024-03-29T10:40:43.694 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:43.702+0000 7f719c379640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71940aaef0 con 0x7f71940a9d70 2024-03-29T10:40:43.694 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:43.702+0000 7f719c379640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7194145bd0 con 0x7f71941457f0 2024-03-29T10:40:43.694 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:43.702+0000 7f719c379640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7194145eb0 con 0x7f719413f650 2024-03-29T10:40:43.694 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:43.702+0000 7f719c379640 10 monclient(hunting): _renew_subs 2024-03-29T10:40:43.694 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:43.702+0000 7f719c379640 10 monclient(hunting): authenticate will time out at 765.938543s 2024-03-29T10:40:43.694 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:43.702+0000 7f719a916640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f71940a9d70 0x7f719414d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:58754/0 (socket says 172.21.15.88:58754) 2024-03-29T10:40:43.694 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:43.702+0000 7f7199914640 1 --1- >> v1:172.21.15.88:6790/0 conn(0x7f719413f650 0x7f71940b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6790/0 says I am v1:172.21.15.88:35650/0 (socket says 172.21.15.88:35650) 2024-03-29T10:40:43.694 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:43.702+0000 7f719a916640 1 -- 172.21.15.88:0/975128263 learned_addr learned my addr 172.21.15.88:0/975128263 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-29T10:40:43.694 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:43.702+0000 7f7199113640 1 -- 172.21.15.88:0/975128263 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4066638463 0 0) 0x7f71940aaef0 con 0x7f71940a9d70 2024-03-29T10:40:43.694 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:43.703+0000 7f7199113640 10 monclient(hunting): _negotiate 2024-03-29T10:40:43.695 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:43.703+0000 7f7199113640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:43.695 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:43.703+0000 7f7199113640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:43.695 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:43.703+0000 7f7199113640 10 monclient(hunting): my global_id is 4188 2024-03-29T10:40:43.695 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:43.703+0000 7f7199113640 1 -- 172.21.15.88:0/975128263 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7180003520 con 0x7f71940a9d70 2024-03-29T10:40:43.695 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:43.703+0000 7f7199113640 1 -- 172.21.15.88:0/975128263 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1988594670 0 0) 0x7f7194145eb0 con 0x7f719413f650 2024-03-29T10:40:43.695 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:43.703+0000 7f7199113640 10 monclient(hunting): _negotiate 2024-03-29T10:40:43.695 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:43.703+0000 7f7199113640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:43.695 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:43.703+0000 7f7199113640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:43.695 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:43.703+0000 7f7199113640 10 monclient(hunting): my global_id is 4184 2024-03-29T10:40:43.695 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:43.703+0000 7f7199113640 1 -- 172.21.15.88:0/975128263 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f71940aaef0 con 0x7f719413f650 2024-03-29T10:40:43.695 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:43.703+0000 7f7199113640 1 -- 172.21.15.88:0/975128263 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 632987232 0 0) 0x7f7180003520 con 0x7f71940a9d70 2024-03-29T10:40:43.695 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:43.703+0000 7f7199113640 1 -- 172.21.15.88:0/975128263 >> v1:172.21.15.88:6790/0 conn(0x7f719413f650 legacy=0x7f71940b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:43.695 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:43.703+0000 7f7199113640 1 -- 172.21.15.88:0/975128263 >> v1:172.21.15.196:6789/0 conn(0x7f71941457f0 legacy=0x7f719413ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:43.695 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:43.703+0000 7f7199113640 10 monclient: _finish_hunting 0 2024-03-29T10:40:43.695 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:43.703+0000 7f7199113640 1 monclient: found mon.noname-a 2024-03-29T10:40:43.696 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:43.703+0000 7f7199113640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:40:43.696 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:43.703+0000 7f7199113640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.88:6789/0 2024-03-29T10:40:43.696 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:43.703+0000 7f7199113640 1 -- 172.21.15.88:0/975128263 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7194146340 con 0x7f71940a9d70 2024-03-29T10:40:43.696 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:43.703+0000 7f7199113640 10 monclient: _finish_auth 0 2024-03-29T10:40:43.696 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:43.703+0000 7f7199113640 10 monclient: _check_auth_tickets 2024-03-29T10:40:43.696 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:43.703+0000 7f7199113640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:40:43.696 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:43.703+0000 7f7199113640 1 -- 172.21.15.88:0/975128263 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f718c001a10 con 0x7f71940a9d70 2024-03-29T10:40:43.696 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:43.703+0000 7f7199113640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:43.696 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:43.703+0000 7f7199113640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:40:43.696 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:43.703+0000 7f7199113640 10 monclient: dump: 2024-03-29T10:40:43.696 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:40:43.696 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:40:43.696 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:40:43.696 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:40:43.697 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:43.697 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:40:43.697 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:40:43.697 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:40:43.697 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:40:43.697 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:40:43.697 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:43.703+0000 7f719c379640 5 monclient: authenticate success, global_id 4188 2024-03-29T10:40:43.697 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:43.703+0000 7f719c379640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:40:43.697 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:43.703+0000 7f7199113640 1 -- 172.21.15.88:0/975128263 <== mon.0 v1:172.21.15.88:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f718c001bf0 con 0x7f71940a9d70 2024-03-29T10:40:43.697 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:43.703+0000 7f7199113640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:40:43.697 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:43.703+0000 7f719c379640 10 monclient: get_monmap_and_config success 2024-03-29T10:40:43.697 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:43.703+0000 7f7199113640 1 -- 172.21.15.88:0/975128263 <== mon.0 v1:172.21.15.88:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f718c002000 con 0x7f71940a9d70 2024-03-29T10:40:43.697 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:43.703+0000 7f7199113640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:43.697 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:43.703+0000 7f719c379640 4 set_mon_vals no callback set 2024-03-29T10:40:43.697 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:43.703+0000 7f7199113640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:40:43.698 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:43.703+0000 7f7199113640 10 monclient: dump: 2024-03-29T10:40:43.698 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:40:43.698 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:40:43.698 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:40:43.698 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:40:43.698 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:43.698 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:40:43.698 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:40:43.698 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:40:43.698 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:40:43.698 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:40:43.698 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:43.703+0000 7f719c379640 10 monclient: shutdown 2024-03-29T10:40:43.698 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:43.703+0000 7f719c379640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:40:43.698 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:43.703+0000 7f719c379640 1 -- 172.21.15.88:0/975128263 >> v1:172.21.15.88:6789/0 conn(0x7f71940a9d70 legacy=0x7f719414d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:43.698 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:43.703+0000 7f719c379640 1 -- 172.21.15.88:0/975128263 shutdown_connections 2024-03-29T10:40:43.699 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:43.703+0000 7f719c379640 1 -- 172.21.15.88:0/975128263 >> 172.21.15.88:0/975128263 conn(0x7f7194066a00 msgr2=0x7f7194066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:40:43.699 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:43.704+0000 7f719c379640 1 -- 172.21.15.88:0/975128263 shutdown_connections 2024-03-29T10:40:43.699 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:43.704+0000 7f719c379640 1 -- 172.21.15.88:0/975128263 wait complete. 2024-03-29T10:40:43.699 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:43.704+0000 7f719c379640 10 monclient: build_initial_monmap 2024-03-29T10:40:43.699 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:43.704+0000 7f719c379640 10 monclient: monmap: 2024-03-29T10:40:43.699 INFO:teuthology.misc.health.smithi088.stderr:epoch 0 2024-03-29T10:40:43.699 INFO:teuthology.misc.health.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:40:43.699 INFO:teuthology.misc.health.smithi088.stderr:last_changed 0.000000 2024-03-29T10:40:43.699 INFO:teuthology.misc.health.smithi088.stderr:created 0.000000 2024-03-29T10:40:43.699 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 0 (unknown) 2024-03-29T10:40:43.699 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:40:43.699 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-03-29T10:40:43.699 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-29T10:40:43.699 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-c 2024-03-29T10:40:43.699 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:40:43.700 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:43.704+0000 7f719c379640 1 librados: starting msgr at 2024-03-29T10:40:43.700 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:43.704+0000 7f719c379640 1 librados: starting objecter 2024-03-29T10:40:43.700 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:43.704+0000 7f719c379640 1 Processor -- start 2024-03-29T10:40:43.700 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:43.704+0000 7f719c379640 1 -- start start 2024-03-29T10:40:43.700 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:43.704+0000 7f719c379640 1 librados: setting wanted keys 2024-03-29T10:40:43.700 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:43.704+0000 7f719c379640 1 librados: calling monclient init 2024-03-29T10:40:43.700 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:43.704+0000 7f719c379640 10 monclient: init 2024-03-29T10:40:43.700 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:43.705+0000 7f719c379640 10 monclient: _reopen_session rank -1 2024-03-29T10:40:43.700 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:43.705+0000 7f719c379640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:40:43.700 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:43.705+0000 7f719c379640 10 monclient(hunting): picked mon.noname-c con 0x7f71940a9d70 addr v1:172.21.15.88:6790/0 2024-03-29T10:40:43.700 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:43.705+0000 7f719c379640 10 monclient(hunting): picked mon.noname-b con 0x7f719413f650 addr v1:172.21.15.196:6789/0 2024-03-29T10:40:43.700 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:43.705+0000 7f719c379640 10 monclient(hunting): picked mon.noname-a con 0x7f71941457f0 addr v1:172.21.15.88:6789/0 2024-03-29T10:40:43.700 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:43.705+0000 7f719c379640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71941d7240 con 0x7f71941457f0 2024-03-29T10:40:43.701 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:43.705+0000 7f719c379640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71941387a0 con 0x7f719413f650 2024-03-29T10:40:43.701 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:43.705+0000 7f719c379640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71941d80c0 con 0x7f71940a9d70 2024-03-29T10:40:43.701 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:43.705+0000 7f719c379640 10 monclient(hunting): _renew_subs 2024-03-29T10:40:43.701 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:43.705+0000 7f719c379640 10 monclient(hunting): authenticate will time out at 765.941381s 2024-03-29T10:40:43.701 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:43.705+0000 7f719a115640 1 --1- >> v1:172.21.15.88:6790/0 conn(0x7f71940a9d70 0x7f71941d79b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6790/0 says I am v1:172.21.15.88:35666/0 (socket says 172.21.15.88:35666) 2024-03-29T10:40:43.701 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:43.705+0000 7f719a115640 1 -- 172.21.15.88:0/3891167715 learned_addr learned my addr 172.21.15.88:0/3891167715 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-29T10:40:43.702 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:43.705+0000 7f718affd640 10 client.?.objecter ms_handle_connect 0x7f71941457f0 2024-03-29T10:40:43.702 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:43.705+0000 7f718affd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:40:43.702 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:43.705+0000 7f718affd640 10 client.?.objecter ms_handle_connect 0x7f71940a9d70 2024-03-29T10:40:43.702 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:43.705+0000 7f718affd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:40:43.702 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:43.705+0000 7f718affd640 10 client.?.objecter ms_handle_connect 0x7f719413f650 2024-03-29T10:40:43.703 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:43.705+0000 7f718affd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:40:43.703 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:43.705+0000 7f718affd640 1 -- 172.21.15.88:0/3891167715 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3670074146 0 0) 0x7f71941d80c0 con 0x7f71940a9d70 2024-03-29T10:40:43.703 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:43.705+0000 7f718affd640 10 monclient(hunting): _negotiate 2024-03-29T10:40:43.703 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:43.705+0000 7f718affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:43.703 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:43.705+0000 7f718affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:43.703 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:43.705+0000 7f718affd640 10 monclient(hunting): my global_id is 4187 2024-03-29T10:40:43.703 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:43.705+0000 7f718affd640 1 -- 172.21.15.88:0/3891167715 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7178004eb0 con 0x7f71940a9d70 2024-03-29T10:40:43.703 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:43.705+0000 7f718affd640 1 -- 172.21.15.88:0/3891167715 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3761055706 0 0) 0x7f71941d7240 con 0x7f71941457f0 2024-03-29T10:40:43.703 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:43.705+0000 7f718affd640 10 monclient(hunting): _negotiate 2024-03-29T10:40:43.703 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:43.705+0000 7f718affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:43.703 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:43.705+0000 7f718affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:43.703 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:43.705+0000 7f718affd640 10 monclient(hunting): my global_id is 4191 2024-03-29T10:40:43.703 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:43.705+0000 7f718affd640 1 -- 172.21.15.88:0/3891167715 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f71941d80c0 con 0x7f71941457f0 2024-03-29T10:40:43.703 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:43.705+0000 7f718affd640 1 -- 172.21.15.88:0/3891167715 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2097772384 0 0) 0x7f71941387a0 con 0x7f719413f650 2024-03-29T10:40:43.704 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:43.705+0000 7f718affd640 10 monclient(hunting): _negotiate 2024-03-29T10:40:43.704 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:43.705+0000 7f718affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:43.704 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:43.705+0000 7f718affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:43.704 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:43.705+0000 7f718affd640 10 monclient(hunting): my global_id is 4186 2024-03-29T10:40:43.704 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:43.705+0000 7f718affd640 1 -- 172.21.15.88:0/3891167715 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f71941d7240 con 0x7f719413f650 2024-03-29T10:40:43.704 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:43.706+0000 7f718affd640 1 -- 172.21.15.88:0/3891167715 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 359063064 0 0) 0x7f7178004eb0 con 0x7f71940a9d70 2024-03-29T10:40:43.704 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:43.706+0000 7f718affd640 1 -- 172.21.15.88:0/3891167715 --> v1:172.21.15.88:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f71941387a0 con 0x7f71940a9d70 2024-03-29T10:40:43.704 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:43.706+0000 7f718affd640 1 -- 172.21.15.88:0/3891167715 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f7184001990 con 0x7f71940a9d70 2024-03-29T10:40:43.704 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:43.706+0000 7f718affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:43.704 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:43.706+0000 7f718affd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:40:43.704 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:43.706+0000 7f718affd640 10 monclient(hunting): dump: 2024-03-29T10:40:43.704 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:40:43.704 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:40:43.704 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:40:43.705 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:40:43.705 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:43.705 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:40:43.705 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:40:43.705 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:40:43.705 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:40:43.705 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:40:43.705 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:43.706+0000 7f718affd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:40:43.705 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:43.706+0000 7f718affd640 1 -- 172.21.15.88:0/3891167715 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1310993049 0 0) 0x7f71941d80c0 con 0x7f71941457f0 2024-03-29T10:40:43.705 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:43.706+0000 7f718affd640 1 -- 172.21.15.88:0/3891167715 --> v1:172.21.15.88:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7178004eb0 con 0x7f71941457f0 2024-03-29T10:40:43.705 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:43.706+0000 7f718affd640 1 -- 172.21.15.88:0/3891167715 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f718c001d60 con 0x7f71941457f0 2024-03-29T10:40:43.705 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:43.706+0000 7f718affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:43.705 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:43.706+0000 7f718affd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:40:43.705 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:43.706+0000 7f718affd640 10 monclient(hunting): dump: 2024-03-29T10:40:43.705 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:40:43.706 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:40:43.706 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:40:43.706 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:40:43.706 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:43.706 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:40:43.706 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:40:43.706 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:40:43.706 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:40:43.706 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:40:43.706 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:43.706+0000 7f718affd640 1 -- 172.21.15.88:0/3891167715 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2551879288 0 0) 0x7f71941d7240 con 0x7f719413f650 2024-03-29T10:40:43.706 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:43.706+0000 7f718affd640 1 -- 172.21.15.88:0/3891167715 --> v1:172.21.15.196:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f71941d80c0 con 0x7f719413f650 2024-03-29T10:40:43.706 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:43.706+0000 7f718affd640 1 -- 172.21.15.88:0/3891167715 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f7190001760 con 0x7f719413f650 2024-03-29T10:40:43.706 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:43.706+0000 7f718affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:43.706 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:43.706+0000 7f718affd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T10:40:43.707 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:43.706+0000 7f718affd640 10 monclient(hunting): dump: 2024-03-29T10:40:43.707 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:40:43.707 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:40:43.707 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:40:43.707 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:40:43.707 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:43.707 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:40:43.707 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:40:43.707 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:40:43.707 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:40:43.707 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:40:43.707 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:43.706+0000 7f718affd640 1 -- 172.21.15.88:0/3891167715 <== mon.2 v1:172.21.15.88:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3528694935 0 0) 0x7f71941387a0 con 0x7f71940a9d70 2024-03-29T10:40:43.707 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:43.706+0000 7f718affd640 1 -- 172.21.15.88:0/3891167715 >> v1:172.21.15.196:6789/0 conn(0x7f719413f650 legacy=0x7f7194137980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:43.707 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:43.706+0000 7f718affd640 1 -- 172.21.15.88:0/3891167715 >> v1:172.21.15.88:6789/0 conn(0x7f71941457f0 legacy=0x7f7194138090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:43.708 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:43.706+0000 7f718affd640 10 monclient: _finish_hunting 0 2024-03-29T10:40:43.708 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:43.706+0000 7f718affd640 1 monclient: found mon.c 2024-03-29T10:40:43.708 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:43.706+0000 7f718affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:40:43.708 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:43.706+0000 7f718affd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-03-29T10:40:43.708 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:43.706+0000 7f718affd640 1 -- 172.21.15.88:0/3891167715 --> v1:172.21.15.88:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71941d7490 con 0x7f71940a9d70 2024-03-29T10:40:43.708 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:43.706+0000 7f718affd640 10 monclient: _finish_auth 0 2024-03-29T10:40:43.708 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:43.706+0000 7f718affd640 10 monclient: _check_auth_tickets 2024-03-29T10:40:43.708 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:43.706+0000 7f718affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:40:43.708 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:43.706+0000 7f719c379640 5 monclient: authenticate success, global_id 4187 2024-03-29T10:40:43.708 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:43.706+0000 7f719c379640 10 monclient: _renew_subs 2024-03-29T10:40:43.708 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:43.706+0000 7f719c379640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-03-29T10:40:43.708 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:43.706+0000 7f719c379640 1 -- 172.21.15.88:0/3891167715 --> v1:172.21.15.88:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71940ab660 con 0x7f71940a9d70 2024-03-29T10:40:43.708 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:43.706+0000 7f719c379640 10 client.4187.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:40:43.708 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:43.706+0000 7f719c379640 10 monclient: _renew_subs 2024-03-29T10:40:43.709 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:43.706+0000 7f719c379640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-03-29T10:40:43.709 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:43.706+0000 7f719c379640 1 -- 172.21.15.88:0/3891167715 --> v1:172.21.15.88:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f719413ffd0 con 0x7f71940a9d70 2024-03-29T10:40:43.709 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:43.706+0000 7f719c379640 1 librados: init done 2024-03-29T10:40:43.709 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:43.706+0000 7f718affd640 1 -- 172.21.15.88:0/3891167715 <== mon.2 v1:172.21.15.88:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7184001c50 con 0x7f71940a9d70 2024-03-29T10:40:43.709 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:43.706+0000 7f718affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:40:43.709 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:43.706+0000 7f718affd640 1 -- 172.21.15.88:0/3891167715 <== mon.2 v1:172.21.15.88:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f7184003a60 con 0x7f71940a9d70 2024-03-29T10:40:43.709 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:43.706+0000 7f718affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:43.709 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:43.706+0000 7f718affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:40:43.709 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:43.706+0000 7f7198912640 4 set_mon_vals no callback set 2024-03-29T10:40:43.709 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:43.706+0000 7f718affd640 10 monclient: dump: 2024-03-29T10:40:43.709 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:40:43.709 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:40:43.709 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:40:43.709 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:40:43.710 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:43.710 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:40:43.710 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:40:43.710 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:40:43.710 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:40:43.710 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:40:43.710 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:43.706+0000 7f718affd640 1 -- 172.21.15.88:0/3891167715 <== mon.2 v1:172.21.15.88:6790/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f7184004010 con 0x7f71940a9d70 2024-03-29T10:40:43.710 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:43.706+0000 7f718affd640 1 -- 172.21.15.88:0/3891167715 <== mon.2 v1:172.21.15.88:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4257673582 0 0) 0x7f7184006a60 con 0x7f71940a9d70 2024-03-29T10:40:43.710 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:43.706+0000 7f718affd640 10 client.4187.objecter ms_dispatch 0x7f7194000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:40:43.710 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:43.706+0000 7f718affd640 3 client.4187.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:40:43.710 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:43.706+0000 7f718affd640 3 client.4187.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:40:43.710 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:43.706+0000 7f718affd640 20 client.4187.objecter dump_active .. 0 homeless 2024-03-29T10:40:43.710 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:43.707+0000 7f719c379640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:40:43.710 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:43.707+0000 7f719c379640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:40:43.711 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:43.707+0000 7f719c379640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-03-29T10:40:43.711 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:43.707+0000 7f719c379640 1 -- 172.21.15.88:0/3891167715 --> v1:172.21.15.88:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f717c005080 con 0x7f71940a9d70 2024-03-29T10:40:43.711 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:43.709+0000 7f718affd640 1 -- 172.21.15.88:0/3891167715 <== mon.2 v1:172.21.15.88:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f7184006390 con 0x7f71940a9d70 2024-03-29T10:40:43.711 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:43.709+0000 7f718affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:40:43.711 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:43.709+0000 7f718affd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:40:43.754 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:43.769+0000 7f719c379640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:40:43.754 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:43.769+0000 7f719c379640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:40:43.754 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:43.769+0000 7f719c379640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-03-29T10:40:43.754 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:43.769+0000 7f719c379640 1 -- 172.21.15.88:0/3891167715 --> v1:172.21.15.88:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f717c005330 con 0x7f71940a9d70 2024-03-29T10:40:43.754 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:43.770+0000 7f718affd640 1 -- 172.21.15.88:0/3891167715 <== mon.2 v1:172.21.15.88:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 344506374) 0x7f7184006530 con 0x7f71940a9d70 2024-03-29T10:40:43.754 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:43.770+0000 7f718affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:40:43.754 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:43.770+0000 7f718affd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:40:43.754 INFO:teuthology.misc.health.smithi088.stdout: 2024-03-29T10:40:43.755 INFO:teuthology.misc.health.smithi088.stdout:{"epoch":10,"fsid":"869f5188-f92b-42dd-a46d-8c646e0388a9","created":"2024-03-29T10:40:40.285087+0000","modified":"2024-03-29T10:40:42.536230+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:40:42.536230+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a26c0c1c-8de0-438c-a3f6-3df10ea61574","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5de24915-8dec-49b6-8b16-f3170197c591","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ac054af3-d589-4565-9644-e6af75975c53","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3ec4f045-c714-4a22-bd1a-e07330b45ab3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"cddb9867-8e30-4951-9b13-3d1e78b9c45a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dc6d9911-785d-432f-8b53-56a1ccab302f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"77c88437-b587-496e-8b4d-1824ea8b5305","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ce963d96-5a8e-44c3-99de-c0dbebe234ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:40:43.756 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:43.770+0000 7f719c379640 10 librados: watch_flush enter 2024-03-29T10:40:43.756 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:43.770+0000 7f719c379640 10 librados: watch_flush exit 2024-03-29T10:40:43.756 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:43.770+0000 7f719c379640 20 client.4187.objecter shutdown clearing up homeless session... 2024-03-29T10:40:43.756 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:43.770+0000 7f719c379640 10 client.4187.objecter successfully canceled tick 2024-03-29T10:40:43.756 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:43.771+0000 7f719c379640 10 monclient: shutdown 2024-03-29T10:40:43.756 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:43.771+0000 7f719c379640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:40:43.756 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:43.771+0000 7f719c379640 1 -- 172.21.15.88:0/3891167715 >> v1:172.21.15.88:6790/0 conn(0x7f71940a9d70 legacy=0x7f71941d79b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:43.756 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:43.771+0000 7f719c379640 1 -- 172.21.15.88:0/3891167715 shutdown_connections 2024-03-29T10:40:43.756 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:43.771+0000 7f719c379640 1 -- 172.21.15.88:0/3891167715 >> 172.21.15.88:0/3891167715 conn(0x7f7194066a00 msgr2=0x7f71940b5810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:40:43.756 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:43.771+0000 7f719c379640 1 -- 172.21.15.88:0/3891167715 shutdown_connections 2024-03-29T10:40:43.757 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:43.771+0000 7f719c379640 1 -- 172.21.15.88:0/3891167715 wait complete. 2024-03-29T10:40:43.757 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:43.771+0000 7f719c379640 1 librados: shutdown 2024-03-29T10:40:43.763 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:40:44.033 INFO:tasks.ceph.mgr.y.smithi088.stderr:2024-03-29T10:40:44.049+0000 7fca56440040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-29T10:40:44.047 INFO:tasks.ceph.mgr.x.smithi196.stderr:2024-03-29T10:40:44.064+0000 7ff5dcc4b040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-29T10:40:44.305 INFO:tasks.ceph.mgr.y.smithi088.stderr:2024-03-29T10:40:44.321+0000 7fca56440040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-29T10:40:44.320 INFO:tasks.ceph.mgr.x.smithi196.stderr:2024-03-29T10:40:44.336+0000 7ff5dcc4b040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-29T10:40:44.355 INFO:tasks.ceph.mgr.y.smithi088.stderr:2024-03-29T10:40:44.371+0000 7fca56440040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-29T10:40:44.373 INFO:tasks.ceph.mgr.x.smithi196.stderr:2024-03-29T10:40:44.389+0000 7ff5dcc4b040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-29T10:40:44.468 INFO:tasks.ceph.mgr.y.smithi088.stderr:2024-03-29T10:40:44.484+0000 7fca56440040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-29T10:40:44.488 INFO:tasks.ceph.mgr.x.smithi196.stderr:2024-03-29T10:40:44.504+0000 7ff5dcc4b040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-29T10:40:45.484 INFO:tasks.ceph.mgr.y.smithi088.stderr:2024-03-29T10:40:45.500+0000 7fc9f789d640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-03-29T10:40:45.484 INFO:tasks.ceph.mgr.y.smithi088.stderr:2024-03-29T10:40:45.501+0000 7fc9f789d640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-03-29T10:40:46.625 INFO:tasks.ceph.osd.3.smithi088.stderr:2024-03-29T10:40:46.642+0000 7f5366413640 -1 osd.3 0 log_to_monitors true 2024-03-29T10:40:46.678 INFO:tasks.ceph.osd.2.smithi088.stderr:2024-03-29T10:40:46.694+0000 7f2263a0e640 -1 osd.2 0 log_to_monitors true 2024-03-29T10:40:46.678 INFO:tasks.ceph.osd.1.smithi088.stderr:2024-03-29T10:40:46.695+0000 7fa68661c640 -1 osd.1 0 log_to_monitors true 2024-03-29T10:40:46.680 INFO:tasks.ceph.osd.0.smithi088.stderr:2024-03-29T10:40:46.696+0000 7f5811213640 -1 osd.0 0 log_to_monitors true 2024-03-29T10:40:46.705 INFO:tasks.ceph.osd.6.smithi196.stderr:2024-03-29T10:40:46.722+0000 7f055be68640 -1 osd.6 0 log_to_monitors true 2024-03-29T10:40:46.773 INFO:tasks.ceph.osd.7.smithi196.stderr:2024-03-29T10:40:46.790+0000 7fdda0a15640 -1 osd.7 0 log_to_monitors true 2024-03-29T10:40:46.774 INFO:tasks.ceph.osd.4.smithi196.stderr:2024-03-29T10:40:46.790+0000 7f73220f9640 -1 osd.4 0 log_to_monitors true 2024-03-29T10:40:46.786 INFO:tasks.ceph.osd.5.smithi196.stderr:2024-03-29T10:40:46.802+0000 7fc4a100c640 -1 osd.5 0 log_to_monitors true 2024-03-29T10:40:50.569 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:40:50.732 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:50.748+0000 7ff3a69c8640 10 monclient: get_monmap_and_config 2024-03-29T10:40:50.732 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:50.748+0000 7ff3a69c8640 10 monclient: build_initial_monmap 2024-03-29T10:40:50.732 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:50.748+0000 7ff3a69c8640 10 monclient: monmap: 2024-03-29T10:40:50.732 INFO:teuthology.misc.health.smithi088.stderr:epoch 0 2024-03-29T10:40:50.732 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:40:50.732 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:50.749088+0000 2024-03-29T10:40:50.732 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:50.749088+0000 2024-03-29T10:40:50.732 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 0 (unknown) 2024-03-29T10:40:50.732 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:40:50.732 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-03-29T10:40:50.732 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-29T10:40:50.733 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-b 2024-03-29T10:40:50.733 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:40:50.733 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:50.749+0000 7ff3a69c8640 1 Processor -- start 2024-03-29T10:40:50.733 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:50.749+0000 7ff3a69c8640 1 -- start start 2024-03-29T10:40:50.733 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:50.749+0000 7ff3a69c8640 10 monclient: init 2024-03-29T10:40:50.739 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:50.750+0000 7ff3a69c8640 10 monclient: _reopen_session rank -1 2024-03-29T10:40:50.739 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:50.750+0000 7ff3a69c8640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:40:50.739 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:50.750+0000 7ff3a69c8640 10 monclient(hunting): picked mon.noname-a con 0x7ff3a01422b0 addr v1:172.21.15.88:6789/0 2024-03-29T10:40:50.739 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:50.750+0000 7ff3a69c8640 10 monclient(hunting): picked mon.noname-c con 0x7ff3a0142690 addr v1:172.21.15.196:6789/0 2024-03-29T10:40:50.739 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:50.750+0000 7ff3a69c8640 10 monclient(hunting): picked mon.noname-b con 0x7ff3a00b17d0 addr v1:172.21.15.88:6790/0 2024-03-29T10:40:50.740 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:50.750+0000 7ff3a69c8640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3a0142c80 con 0x7ff3a01422b0 2024-03-29T10:40:50.740 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:50.750+0000 7ff3a69c8640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3a00aae40 con 0x7ff3a0142690 2024-03-29T10:40:50.740 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:50.750+0000 7ff3a69c8640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3a013fbd0 con 0x7ff3a00b17d0 2024-03-29T10:40:50.740 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:50.750+0000 7ff3a69c8640 10 monclient(hunting): _renew_subs 2024-03-29T10:40:50.740 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:50.750+0000 7ff3a69c8640 10 monclient(hunting): authenticate will time out at 772.986543s 2024-03-29T10:40:50.740 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:50.750+0000 7ff3a4f65640 1 --1- >> v1:172.21.15.88:6790/0 conn(0x7ff3a00b17d0 0x7ff3a00b2ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6790/0 says I am v1:172.21.15.88:39776/0 (socket says 172.21.15.88:39776) 2024-03-29T10:40:50.740 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:50.750+0000 7ff3a4f65640 1 -- 172.21.15.88:0/3840186825 learned_addr learned my addr 172.21.15.88:0/3840186825 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-29T10:40:50.740 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:50.751+0000 7ff39effd640 1 -- 172.21.15.88:0/3840186825 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4134183011 0 0) 0x7ff3a013fbd0 con 0x7ff3a00b17d0 2024-03-29T10:40:50.740 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:50.751+0000 7ff39effd640 10 monclient(hunting): _negotiate 2024-03-29T10:40:50.740 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:50.751+0000 7ff39effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:50.740 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:50.751+0000 7ff39effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:50.740 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:50.751+0000 7ff39effd640 10 monclient(hunting): my global_id is 4229 2024-03-29T10:40:50.740 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:50.751+0000 7ff39effd640 1 -- 172.21.15.88:0/3840186825 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff388003520 con 0x7ff3a00b17d0 2024-03-29T10:40:50.741 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:50.751+0000 7ff39effd640 1 -- 172.21.15.88:0/3840186825 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3387589319 0 0) 0x7ff3a00aae40 con 0x7ff3a0142690 2024-03-29T10:40:50.741 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:50.751+0000 7ff39effd640 10 monclient(hunting): _negotiate 2024-03-29T10:40:50.741 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:50.751+0000 7ff39effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:50.741 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:50.751+0000 7ff39effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:50.741 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:50.752+0000 7ff39effd640 10 monclient(hunting): my global_id is 4228 2024-03-29T10:40:50.741 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:50.752+0000 7ff39effd640 1 -- 172.21.15.88:0/3840186825 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3a013fbd0 con 0x7ff3a0142690 2024-03-29T10:40:50.741 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:50.752+0000 7ff39effd640 1 -- 172.21.15.88:0/3840186825 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2663740525 0 0) 0x7ff3a0142c80 con 0x7ff3a01422b0 2024-03-29T10:40:50.741 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:50.752+0000 7ff39effd640 10 monclient(hunting): _negotiate 2024-03-29T10:40:50.741 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:50.752+0000 7ff39effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:50.741 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:50.752+0000 7ff39effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:50.741 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:50.752+0000 7ff39effd640 10 monclient(hunting): my global_id is 4236 2024-03-29T10:40:50.741 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:50.752+0000 7ff39effd640 1 -- 172.21.15.88:0/3840186825 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3a00aae40 con 0x7ff3a01422b0 2024-03-29T10:40:50.741 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:50.752+0000 7ff39effd640 1 -- 172.21.15.88:0/3840186825 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3203941017 0 0) 0x7ff3a013fbd0 con 0x7ff3a0142690 2024-03-29T10:40:50.741 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:50.752+0000 7ff39effd640 1 -- 172.21.15.88:0/3840186825 >> v1:172.21.15.88:6790/0 conn(0x7ff3a00b17d0 legacy=0x7ff3a00b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:50.742 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:50.752+0000 7ff39effd640 1 -- 172.21.15.88:0/3840186825 >> v1:172.21.15.88:6789/0 conn(0x7ff3a01422b0 legacy=0x7ff3a013f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:50.742 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:50.752+0000 7ff39effd640 10 monclient: _finish_hunting 0 2024-03-29T10:40:50.742 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:50.752+0000 7ff39effd640 1 monclient: found mon.noname-c 2024-03-29T10:40:50.742 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:50.752+0000 7ff39effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:40:50.742 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:50.752+0000 7ff39effd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.196:6789/0 2024-03-29T10:40:50.742 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:50.752+0000 7ff39effd640 1 -- 172.21.15.88:0/3840186825 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3a0151940 con 0x7ff3a0142690 2024-03-29T10:40:50.742 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:50.752+0000 7ff39effd640 10 monclient: _finish_auth 0 2024-03-29T10:40:50.742 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:50.752+0000 7ff39effd640 10 monclient: _check_auth_tickets 2024-03-29T10:40:50.742 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:50.752+0000 7ff39effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:40:50.742 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:50.753+0000 7ff3a69c8640 5 monclient: authenticate success, global_id 4228 2024-03-29T10:40:50.742 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:50.753+0000 7ff3a69c8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:40:50.742 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:50.753+0000 7ff39effd640 1 -- 172.21.15.88:0/3840186825 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7ff3940015a0 con 0x7ff3a0142690 2024-03-29T10:40:50.742 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:50.753+0000 7ff39effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:50.742 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:50.753+0000 7ff39effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:40:50.743 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:50.753+0000 7ff39effd640 10 monclient: dump: 2024-03-29T10:40:50.743 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:40:50.743 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:40:50.743 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:40:50.743 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:40:50.743 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:50.743 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:40:50.743 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:40:50.743 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:40:50.743 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:40:50.743 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:40:50.743 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:50.753+0000 7ff3a69c8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:40:50.743 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:50.753+0000 7ff39effd640 1 -- 172.21.15.88:0/3840186825 <== mon.1 v1:172.21.15.196:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff3940017c0 con 0x7ff3a0142690 2024-03-29T10:40:50.744 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:50.753+0000 7ff39effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:40:50.744 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:50.753+0000 7ff3a69c8640 10 monclient: get_monmap_and_config success 2024-03-29T10:40:50.744 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:50.753+0000 7ff3a69c8640 4 set_mon_vals no callback set 2024-03-29T10:40:50.744 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:50.753+0000 7ff39effd640 1 -- 172.21.15.88:0/3840186825 <== mon.1 v1:172.21.15.196:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7ff394001ad0 con 0x7ff3a0142690 2024-03-29T10:40:50.744 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:50.753+0000 7ff39effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:50.744 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:50.753+0000 7ff39effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:40:50.744 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:50.753+0000 7ff39effd640 10 monclient: dump: 2024-03-29T10:40:50.744 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:40:50.745 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:40:50.745 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:40:50.745 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:40:50.745 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:50.745 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:40:50.745 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:40:50.745 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:40:50.745 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:40:50.745 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:40:50.745 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:50.754+0000 7ff3a69c8640 10 monclient: shutdown 2024-03-29T10:40:50.745 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:50.754+0000 7ff3a69c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:40:50.745 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:50.754+0000 7ff3a69c8640 1 -- 172.21.15.88:0/3840186825 >> v1:172.21.15.196:6789/0 conn(0x7ff3a0142690 legacy=0x7ff3a00b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:50.745 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:50.755+0000 7ff3a69c8640 1 -- 172.21.15.88:0/3840186825 shutdown_connections 2024-03-29T10:40:50.745 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:50.755+0000 7ff3a69c8640 1 -- 172.21.15.88:0/3840186825 >> 172.21.15.88:0/3840186825 conn(0x7ff3a0066a00 msgr2=0x7ff3a0066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:40:50.746 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:50.755+0000 7ff3a69c8640 1 -- 172.21.15.88:0/3840186825 shutdown_connections 2024-03-29T10:40:50.746 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:50.755+0000 7ff3a69c8640 1 -- 172.21.15.88:0/3840186825 wait complete. 2024-03-29T10:40:50.751 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:50.756+0000 7ff3a69c8640 10 monclient: build_initial_monmap 2024-03-29T10:40:50.752 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:50.756+0000 7ff3a69c8640 10 monclient: monmap: 2024-03-29T10:40:50.752 INFO:teuthology.misc.health.smithi088.stderr:epoch 0 2024-03-29T10:40:50.752 INFO:teuthology.misc.health.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:40:50.752 INFO:teuthology.misc.health.smithi088.stderr:last_changed 0.000000 2024-03-29T10:40:50.752 INFO:teuthology.misc.health.smithi088.stderr:created 0.000000 2024-03-29T10:40:50.752 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 0 (unknown) 2024-03-29T10:40:50.752 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:40:50.752 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-03-29T10:40:50.752 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-29T10:40:50.752 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-c 2024-03-29T10:40:50.752 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:40:50.752 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:50.757+0000 7ff3a69c8640 1 librados: starting msgr at 2024-03-29T10:40:50.752 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:50.757+0000 7ff3a69c8640 1 librados: starting objecter 2024-03-29T10:40:50.752 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:50.757+0000 7ff3a69c8640 1 Processor -- start 2024-03-29T10:40:50.753 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:50.758+0000 7ff3a69c8640 1 -- start start 2024-03-29T10:40:50.753 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:50.758+0000 7ff3a69c8640 1 librados: setting wanted keys 2024-03-29T10:40:50.753 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:50.758+0000 7ff3a69c8640 1 librados: calling monclient init 2024-03-29T10:40:50.753 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:50.758+0000 7ff3a69c8640 10 monclient: init 2024-03-29T10:40:50.753 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:50.759+0000 7ff3a69c8640 10 monclient: _reopen_session rank -1 2024-03-29T10:40:50.753 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:50.759+0000 7ff3a69c8640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:40:50.753 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:50.759+0000 7ff39ffff640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7ff3a00b17d0 0x7ff3a012ff00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:51786/0 (socket says 172.21.15.88:51786) 2024-03-29T10:40:50.753 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:50.759+0000 7ff39ffff640 1 -- 172.21.15.88:0/1845046767 learned_addr learned my addr 172.21.15.88:0/1845046767 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-29T10:40:50.753 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:50.759+0000 7ff3a69c8640 10 monclient(hunting): picked mon.noname-a con 0x7ff3a00b17d0 addr v1:172.21.15.88:6789/0 2024-03-29T10:40:50.753 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:50.760+0000 7ff3a69c8640 10 monclient(hunting): picked mon.noname-b con 0x7ff3a01422b0 addr v1:172.21.15.196:6789/0 2024-03-29T10:40:50.753 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:50.760+0000 7ff3a69c8640 10 monclient(hunting): picked mon.noname-c con 0x7ff3a0142690 addr v1:172.21.15.88:6790/0 2024-03-29T10:40:50.753 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:50.760+0000 7ff3a69c8640 1 -- 172.21.15.88:0/1845046767 --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3a00acc30 con 0x7ff3a00b17d0 2024-03-29T10:40:50.753 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:50.760+0000 7ff3a69c8640 1 -- 172.21.15.88:0/1845046767 --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3a0132c70 con 0x7ff3a01422b0 2024-03-29T10:40:50.753 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:50.760+0000 7ff3a69c8640 1 -- 172.21.15.88:0/1845046767 --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3a0132940 con 0x7ff3a0142690 2024-03-29T10:40:50.754 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:50.760+0000 7ff3a69c8640 10 monclient(hunting): _renew_subs 2024-03-29T10:40:50.754 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:50.760+0000 7ff3a69c8640 10 monclient(hunting): authenticate will time out at 772.996351s 2024-03-29T10:40:50.754 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:50.760+0000 7ff39cff9640 10 client.?.objecter ms_handle_connect 0x7ff3a00b17d0 2024-03-29T10:40:50.754 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:50.760+0000 7ff39cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:40:50.754 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:50.760+0000 7ff39cff9640 10 client.?.objecter ms_handle_connect 0x7ff3a01422b0 2024-03-29T10:40:50.754 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:50.760+0000 7ff39cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:40:50.754 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:50.760+0000 7ff39cff9640 10 client.?.objecter ms_handle_connect 0x7ff3a0142690 2024-03-29T10:40:50.754 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:50.760+0000 7ff39cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:40:50.754 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:50.760+0000 7ff39cff9640 1 -- 172.21.15.88:0/1845046767 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 105650421 0 0) 0x7ff3a0132c70 con 0x7ff3a01422b0 2024-03-29T10:40:50.754 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:50.760+0000 7ff39cff9640 10 monclient(hunting): _negotiate 2024-03-29T10:40:50.755 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:50.760+0000 7ff39cff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:50.755 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:50.760+0000 7ff39cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:50.755 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:50.760+0000 7ff39cff9640 10 monclient(hunting): my global_id is 4231 2024-03-29T10:40:50.755 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:50.761+0000 7ff39cff9640 1 -- 172.21.15.88:0/1845046767 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff38c004eb0 con 0x7ff3a01422b0 2024-03-29T10:40:50.755 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:50.761+0000 7ff39cff9640 1 -- 172.21.15.88:0/1845046767 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 177632412 0 0) 0x7ff3a00acc30 con 0x7ff3a00b17d0 2024-03-29T10:40:50.755 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:50.761+0000 7ff39cff9640 10 monclient(hunting): _negotiate 2024-03-29T10:40:50.755 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:50.761+0000 7ff39cff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:50.755 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:50.761+0000 7ff39cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:50.755 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:50.761+0000 7ff39cff9640 10 monclient(hunting): my global_id is 4239 2024-03-29T10:40:50.755 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:50.761+0000 7ff39cff9640 1 -- 172.21.15.88:0/1845046767 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3a0132c70 con 0x7ff3a00b17d0 2024-03-29T10:40:50.755 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:50.761+0000 7ff39cff9640 1 -- 172.21.15.88:0/1845046767 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3011146085 0 0) 0x7ff3a0132940 con 0x7ff3a0142690 2024-03-29T10:40:50.756 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:50.761+0000 7ff39cff9640 10 monclient(hunting): _negotiate 2024-03-29T10:40:50.756 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:50.761+0000 7ff39cff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:50.756 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:50.761+0000 7ff39cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:50.756 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:50.761+0000 7ff39cff9640 10 monclient(hunting): my global_id is 4232 2024-03-29T10:40:50.756 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:50.761+0000 7ff39cff9640 1 -- 172.21.15.88:0/1845046767 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3a00acc30 con 0x7ff3a0142690 2024-03-29T10:40:50.756 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:50.761+0000 7ff39cff9640 1 -- 172.21.15.88:0/1845046767 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3353116814 0 0) 0x7ff3a0132c70 con 0x7ff3a00b17d0 2024-03-29T10:40:50.756 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:50.761+0000 7ff39cff9640 1 -- 172.21.15.88:0/1845046767 --> v1:172.21.15.88:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff3a0132940 con 0x7ff3a00b17d0 2024-03-29T10:40:50.757 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:50.761+0000 7ff39cff9640 1 -- 172.21.15.88:0/1845046767 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7ff398001cb0 con 0x7ff3a00b17d0 2024-03-29T10:40:50.757 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:50.761+0000 7ff39cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:50.757 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:50.761+0000 7ff39cff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:40:50.757 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:50.761+0000 7ff39cff9640 10 monclient(hunting): dump: 2024-03-29T10:40:50.757 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:40:50.757 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:40:50.757 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:40:50.757 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:40:50.757 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:50.757 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:40:50.757 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:40:50.757 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:40:50.757 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:40:50.758 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:40:50.758 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:50.761+0000 7ff39cff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:40:50.758 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:50.761+0000 7ff39cff9640 1 -- 172.21.15.88:0/1845046767 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2956217803 0 0) 0x7ff38c004eb0 con 0x7ff3a01422b0 2024-03-29T10:40:50.758 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:50.761+0000 7ff39cff9640 1 -- 172.21.15.88:0/1845046767 --> v1:172.21.15.196:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff3a0132c70 con 0x7ff3a01422b0 2024-03-29T10:40:50.758 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:50.761+0000 7ff39cff9640 1 -- 172.21.15.88:0/1845046767 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7ff394001ad0 con 0x7ff3a01422b0 2024-03-29T10:40:50.758 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:50.762+0000 7ff39cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:50.758 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:50.762+0000 7ff39cff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T10:40:50.758 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:50.762+0000 7ff39cff9640 10 monclient(hunting): dump: 2024-03-29T10:40:50.758 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:40:50.758 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:40:50.758 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:40:50.758 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:40:50.758 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:50.758 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:40:50.759 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:40:50.759 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:40:50.759 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:40:50.759 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:40:50.759 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:50.762+0000 7ff39cff9640 1 -- 172.21.15.88:0/1845046767 <== mon.0 v1:172.21.15.88:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1197047057 0 0) 0x7ff3a0132940 con 0x7ff3a00b17d0 2024-03-29T10:40:50.759 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:50.762+0000 7ff39cff9640 1 -- 172.21.15.88:0/1845046767 >> v1:172.21.15.88:6790/0 conn(0x7ff3a0142690 legacy=0x7ff3a01e1cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:50.759 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:50.762+0000 7ff39cff9640 1 -- 172.21.15.88:0/1845046767 >> v1:172.21.15.196:6789/0 conn(0x7ff3a01422b0 legacy=0x7ff3a0130610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:50.759 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:50.762+0000 7ff39cff9640 10 monclient: _finish_hunting 0 2024-03-29T10:40:50.759 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:50.762+0000 7ff39cff9640 1 monclient: found mon.a 2024-03-29T10:40:50.759 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:50.762+0000 7ff39cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:40:50.759 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:50.762+0000 7ff39cff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-03-29T10:40:50.759 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:50.762+0000 7ff39cff9640 1 -- 172.21.15.88:0/1845046767 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3a00ace80 con 0x7ff3a00b17d0 2024-03-29T10:40:50.759 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:50.762+0000 7ff39cff9640 10 monclient: _finish_auth 0 2024-03-29T10:40:50.759 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:50.762+0000 7ff39cff9640 10 monclient: _check_auth_tickets 2024-03-29T10:40:50.760 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:50.762+0000 7ff39cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:40:50.760 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:50.762+0000 7ff3a69c8640 5 monclient: authenticate success, global_id 4239 2024-03-29T10:40:50.760 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:50.762+0000 7ff3a69c8640 10 monclient: _renew_subs 2024-03-29T10:40:50.760 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:50.762+0000 7ff3a69c8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-03-29T10:40:50.760 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:50.762+0000 7ff3a69c8640 1 -- 172.21.15.88:0/1845046767 --> v1:172.21.15.88:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3a00ad160 con 0x7ff3a00b17d0 2024-03-29T10:40:50.760 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:50.762+0000 7ff3a69c8640 10 client.4239.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:40:50.760 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:50.762+0000 7ff3a69c8640 10 monclient: _renew_subs 2024-03-29T10:40:50.760 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:50.762+0000 7ff3a69c8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-03-29T10:40:50.760 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:50.762+0000 7ff3a69c8640 1 -- 172.21.15.88:0/1845046767 --> v1:172.21.15.88:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3a00abfb0 con 0x7ff3a00b17d0 2024-03-29T10:40:50.760 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:50.762+0000 7ff3a69c8640 1 librados: init done 2024-03-29T10:40:50.760 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:50.764+0000 7ff39cff9640 1 -- 172.21.15.88:0/1845046767 <== mon.0 v1:172.21.15.88:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff398003990 con 0x7ff3a00b17d0 2024-03-29T10:40:50.760 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:50.764+0000 7ff39cff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:40:50.760 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:50.764+0000 7ff39cff9640 1 -- 172.21.15.88:0/1845046767 <== mon.0 v1:172.21.15.88:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7ff398003d20 con 0x7ff3a00b17d0 2024-03-29T10:40:50.760 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:50.764+0000 7ff39e7fc640 4 set_mon_vals no callback set 2024-03-29T10:40:50.761 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:50.764+0000 7ff39cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:50.761 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:50.764+0000 7ff39cff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:40:50.761 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:50.764+0000 7ff39cff9640 10 monclient: dump: 2024-03-29T10:40:50.761 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:40:50.761 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:40:50.761 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:40:50.761 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:40:50.761 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:50.761 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:40:50.761 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:40:50.761 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:40:50.761 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:40:50.761 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:40:50.761 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:50.765+0000 7ff3a69c8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:40:50.761 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:50.766+0000 7ff3a69c8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:40:50.762 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:50.766+0000 7ff3a69c8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-03-29T10:40:50.762 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:50.766+0000 7ff3a69c8640 1 -- 172.21.15.88:0/1845046767 --> v1:172.21.15.88:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff380005080 con 0x7ff3a00b17d0 2024-03-29T10:40:50.762 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:50.767+0000 7ff39cff9640 1 -- 172.21.15.88:0/1845046767 <== mon.0 v1:172.21.15.88:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4235629220 0 0) 0x7ff398003e80 con 0x7ff3a00b17d0 2024-03-29T10:40:50.763 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:50.769+0000 7ff39f7fe640 10 monclient: get_auth_request con 0x7ff38c06e030 auth_method 0 2024-03-29T10:40:50.763 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:50.769+0000 7ff39cff9640 1 -- 172.21.15.88:0/1845046767 <== mon.0 v1:172.21.15.88:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4257673582 0 0) 0x7ff398084d20 con 0x7ff3a00b17d0 2024-03-29T10:40:50.763 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:50.769+0000 7ff39cff9640 10 client.4239.objecter ms_dispatch 0x7ff3a0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:40:50.763 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:50.769+0000 7ff39cff9640 3 client.4239.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:40:50.763 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:50.769+0000 7ff39cff9640 3 client.4239.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:40:50.763 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:50.769+0000 7ff39cff9640 20 client.4239.objecter dump_active .. 0 homeless 2024-03-29T10:40:50.763 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:50.769+0000 7ff39cff9640 10 client.4239.objecter ms_handle_connect 0x7ff38c06e030 2024-03-29T10:40:50.764 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:50.778+0000 7ff39cff9640 1 -- 172.21.15.88:0/1845046767 <== mon.0 v1:172.21.15.88:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff398050600 con 0x7ff3a00b17d0 2024-03-29T10:40:50.764 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:50.778+0000 7ff39cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:40:50.764 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:50.778+0000 7ff39cff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:40:51.127 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:51.143+0000 7ff3a69c8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:40:51.127 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:51.143+0000 7ff3a69c8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:40:51.127 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:51.143+0000 7ff3a69c8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-03-29T10:40:51.127 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:51.143+0000 7ff3a69c8640 1 -- 172.21.15.88:0/1845046767 --> v1:172.21.15.88:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff380005330 con 0x7ff3a00b17d0 2024-03-29T10:40:51.128 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:51.145+0000 7ff39cff9640 1 -- 172.21.15.88:0/1845046767 <== mon.0 v1:172.21.15.88:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 344506374) 0x7ff398052530 con 0x7ff3a00b17d0 2024-03-29T10:40:51.129 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:51.145+0000 7ff39cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:40:51.129 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:51.145+0000 7ff39cff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:40:51.129 INFO:teuthology.misc.health.smithi088.stdout: 2024-03-29T10:40:51.129 INFO:teuthology.misc.health.smithi088.stdout:{"epoch":10,"fsid":"869f5188-f92b-42dd-a46d-8c646e0388a9","created":"2024-03-29T10:40:40.285087+0000","modified":"2024-03-29T10:40:42.536230+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:40:42.536230+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a26c0c1c-8de0-438c-a3f6-3df10ea61574","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5de24915-8dec-49b6-8b16-f3170197c591","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ac054af3-d589-4565-9644-e6af75975c53","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3ec4f045-c714-4a22-bd1a-e07330b45ab3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"cddb9867-8e30-4951-9b13-3d1e78b9c45a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dc6d9911-785d-432f-8b53-56a1ccab302f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"77c88437-b587-496e-8b4d-1824ea8b5305","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ce963d96-5a8e-44c3-99de-c0dbebe234ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:40:51.133 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:51.148+0000 7ff3a69c8640 10 librados: watch_flush enter 2024-03-29T10:40:51.133 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:51.148+0000 7ff3a69c8640 10 librados: watch_flush exit 2024-03-29T10:40:51.133 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:51.148+0000 7ff3a69c8640 20 client.4239.objecter shutdown clearing up homeless session... 2024-03-29T10:40:51.133 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:51.148+0000 7ff3a69c8640 10 client.4239.objecter successfully canceled tick 2024-03-29T10:40:51.133 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:51.149+0000 7ff3a69c8640 1 -- 172.21.15.88:0/1845046767 >> v1:172.21.15.88:6816/1752861349 conn(0x7ff38c06e030 legacy=0x7ff38c0704f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:51.133 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:51.149+0000 7ff3a69c8640 10 monclient: shutdown 2024-03-29T10:40:51.133 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:51.149+0000 7ff3a69c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:40:51.133 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:51.149+0000 7ff3a69c8640 1 -- 172.21.15.88:0/1845046767 >> v1:172.21.15.88:6789/0 conn(0x7ff3a00b17d0 legacy=0x7ff3a012ff00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:51.133 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:51.149+0000 7ff3a69c8640 1 -- 172.21.15.88:0/1845046767 shutdown_connections 2024-03-29T10:40:51.133 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:51.149+0000 7ff3a69c8640 1 -- 172.21.15.88:0/1845046767 >> 172.21.15.88:0/1845046767 conn(0x7ff3a0066a00 msgr2=0x7ff3a013db00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:40:51.134 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:51.149+0000 7ff3a69c8640 1 -- 172.21.15.88:0/1845046767 shutdown_connections 2024-03-29T10:40:51.134 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:51.149+0000 7ff3a69c8640 1 -- 172.21.15.88:0/1845046767 wait complete. 2024-03-29T10:40:51.134 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:51.149+0000 7ff3a69c8640 1 librados: shutdown 2024-03-29T10:40:51.148 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:40:57.956 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:40:58.097 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:58.113+0000 7faa449f4640 10 monclient: get_monmap_and_config 2024-03-29T10:40:58.098 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:58.113+0000 7faa449f4640 10 monclient: build_initial_monmap 2024-03-29T10:40:58.098 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:58.114+0000 7faa449f4640 10 monclient: monmap: 2024-03-29T10:40:58.098 INFO:teuthology.misc.health.smithi088.stderr:epoch 0 2024-03-29T10:40:58.098 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:40:58.098 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:58.115025+0000 2024-03-29T10:40:58.098 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:58.115025+0000 2024-03-29T10:40:58.098 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 0 (unknown) 2024-03-29T10:40:58.098 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:40:58.098 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-03-29T10:40:58.098 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-29T10:40:58.098 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-b 2024-03-29T10:40:58.098 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:40:58.103 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:58.114+0000 7faa449f4640 1 Processor -- start 2024-03-29T10:40:58.104 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:58.114+0000 7faa449f4640 1 -- start start 2024-03-29T10:40:58.104 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:58.114+0000 7faa449f4640 10 monclient: init 2024-03-29T10:40:58.104 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:58.115+0000 7faa449f4640 10 monclient: _reopen_session rank -1 2024-03-29T10:40:58.104 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:58.115+0000 7faa449f4640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:40:58.104 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:58.115+0000 7faa449f4640 10 monclient(hunting): picked mon.noname-c con 0x7faa4006ade0 addr v1:172.21.15.196:6789/0 2024-03-29T10:40:58.104 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:58.116+0000 7faa449f4640 10 monclient(hunting): picked mon.noname-a con 0x7faa4013d830 addr v1:172.21.15.88:6789/0 2024-03-29T10:40:58.104 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:58.116+0000 7faa449f4640 10 monclient(hunting): picked mon.noname-b con 0x7faa401405a0 addr v1:172.21.15.88:6790/0 2024-03-29T10:40:58.104 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:58.116+0000 7faa449f4640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa40072290 con 0x7faa4013d830 2024-03-29T10:40:58.104 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:58.116+0000 7faa449f4640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa40071f60 con 0x7faa4006ade0 2024-03-29T10:40:58.104 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:58.116+0000 7faa449f4640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa400b0e00 con 0x7faa401405a0 2024-03-29T10:40:58.104 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:58.116+0000 7faa449f4640 10 monclient(hunting): _renew_subs 2024-03-29T10:40:58.104 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:58.116+0000 7faa449f4640 10 monclient(hunting): authenticate will time out at 780.351956s 2024-03-29T10:40:58.105 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:58.116+0000 7faa3e59c640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7faa4006ade0 0x7faa4013d120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.88:44296/0 (socket says 172.21.15.88:44296) 2024-03-29T10:40:58.105 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:58.116+0000 7faa3e59c640 1 -- 172.21.15.88:0/939936297 learned_addr learned my addr 172.21.15.88:0/939936297 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-29T10:40:58.105 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:58.116+0000 7faa3d59a640 1 -- 172.21.15.88:0/939936297 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4025470532 0 0) 0x7faa40072290 con 0x7faa4013d830 2024-03-29T10:40:58.105 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:58.116+0000 7faa3d59a640 10 monclient(hunting): _negotiate 2024-03-29T10:40:58.105 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:58.116+0000 7faa3d59a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:58.105 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:58.116+0000 7faa3d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:58.105 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:58.116+0000 7faa3d59a640 10 monclient(hunting): my global_id is 4242 2024-03-29T10:40:58.105 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:58.116+0000 7faa3d59a640 1 -- 172.21.15.88:0/939936297 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa28003520 con 0x7faa4013d830 2024-03-29T10:40:58.105 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:58.116+0000 7faa3d59a640 1 -- 172.21.15.88:0/939936297 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1700147624 0 0) 0x7faa400b0e00 con 0x7faa401405a0 2024-03-29T10:40:58.105 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:58.116+0000 7faa3d59a640 10 monclient(hunting): _negotiate 2024-03-29T10:40:58.105 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:58.116+0000 7faa3d59a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:58.105 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:58.116+0000 7faa3d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:58.105 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:58.116+0000 7faa3d59a640 10 monclient(hunting): my global_id is 4235 2024-03-29T10:40:58.105 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:58.116+0000 7faa3d59a640 1 -- 172.21.15.88:0/939936297 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa40072290 con 0x7faa401405a0 2024-03-29T10:40:58.106 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:58.117+0000 7faa3d59a640 1 -- 172.21.15.88:0/939936297 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2533990569 0 0) 0x7faa28003520 con 0x7faa4013d830 2024-03-29T10:40:58.106 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:58.117+0000 7faa3d59a640 1 -- 172.21.15.88:0/939936297 >> v1:172.21.15.88:6790/0 conn(0x7faa401405a0 legacy=0x7faa4014b710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:58.106 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:58.117+0000 7faa3d59a640 1 -- 172.21.15.88:0/939936297 >> v1:172.21.15.196:6789/0 conn(0x7faa4006ade0 legacy=0x7faa4013d120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:58.106 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:58.117+0000 7faa3d59a640 10 monclient: _finish_hunting 0 2024-03-29T10:40:58.106 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:58.117+0000 7faa3d59a640 1 monclient: found mon.noname-a 2024-03-29T10:40:58.106 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:58.117+0000 7faa3d59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:40:58.106 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:58.117+0000 7faa3d59a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.88:6789/0 2024-03-29T10:40:58.106 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:58.117+0000 7faa3d59a640 1 -- 172.21.15.88:0/939936297 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa400af6d0 con 0x7faa4013d830 2024-03-29T10:40:58.106 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:58.117+0000 7faa3d59a640 10 monclient: _finish_auth 0 2024-03-29T10:40:58.106 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:58.117+0000 7faa3d59a640 10 monclient: _check_auth_tickets 2024-03-29T10:40:58.106 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:58.117+0000 7faa3d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:40:58.106 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:58.117+0000 7faa3d59a640 1 -- 172.21.15.88:0/939936297 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7faa34001690 con 0x7faa4013d830 2024-03-29T10:40:58.106 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:58.117+0000 7faa3d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:58.106 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:58.117+0000 7faa3d59a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:40:58.107 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:58.117+0000 7faa3d59a640 10 monclient: dump: 2024-03-29T10:40:58.107 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:40:58.107 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:40:58.107 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:40:58.107 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:40:58.107 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:58.107 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:40:58.107 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:40:58.107 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:40:58.107 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:40:58.107 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:40:58.107 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:58.117+0000 7faa449f4640 5 monclient: authenticate success, global_id 4242 2024-03-29T10:40:58.107 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:58.117+0000 7faa449f4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:40:58.107 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:58.117+0000 7faa3d59a640 1 -- 172.21.15.88:0/939936297 <== mon.0 v1:172.21.15.88:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7faa340017f0 con 0x7faa4013d830 2024-03-29T10:40:58.107 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:58.117+0000 7faa3d59a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:40:58.108 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:58.118+0000 7faa449f4640 10 monclient: get_monmap_and_config success 2024-03-29T10:40:58.108 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:58.118+0000 7faa449f4640 4 set_mon_vals no callback set 2024-03-29T10:40:58.108 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:58.118+0000 7faa3d59a640 1 -- 172.21.15.88:0/939936297 <== mon.0 v1:172.21.15.88:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7faa34001c00 con 0x7faa4013d830 2024-03-29T10:40:58.108 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:58.118+0000 7faa3d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:58.108 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:58.118+0000 7faa3d59a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:40:58.108 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:58.118+0000 7faa3d59a640 10 monclient: dump: 2024-03-29T10:40:58.108 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:40:58.108 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:40:58.108 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:40:58.108 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:40:58.108 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:58.108 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:40:58.108 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:40:58.108 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:40:58.109 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:40:58.109 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:40:58.109 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:58.118+0000 7faa449f4640 10 monclient: shutdown 2024-03-29T10:40:58.109 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:58.118+0000 7faa449f4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:40:58.109 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:58.118+0000 7faa449f4640 1 -- 172.21.15.88:0/939936297 >> v1:172.21.15.88:6789/0 conn(0x7faa4013d830 legacy=0x7faa4013fe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:58.109 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:58.119+0000 7faa449f4640 1 -- 172.21.15.88:0/939936297 shutdown_connections 2024-03-29T10:40:58.109 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:58.119+0000 7faa449f4640 1 -- 172.21.15.88:0/939936297 >> 172.21.15.88:0/939936297 conn(0x7faa400b4930 msgr2=0x7faa400b4d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:40:58.109 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:58.119+0000 7faa449f4640 1 -- 172.21.15.88:0/939936297 shutdown_connections 2024-03-29T10:40:58.109 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:58.119+0000 7faa449f4640 1 -- 172.21.15.88:0/939936297 wait complete. 2024-03-29T10:40:58.112 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:58.120+0000 7faa449f4640 10 monclient: build_initial_monmap 2024-03-29T10:40:58.112 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:58.120+0000 7faa449f4640 10 monclient: monmap: 2024-03-29T10:40:58.112 INFO:teuthology.misc.health.smithi088.stderr:epoch 0 2024-03-29T10:40:58.112 INFO:teuthology.misc.health.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:40:58.113 INFO:teuthology.misc.health.smithi088.stderr:last_changed 0.000000 2024-03-29T10:40:58.113 INFO:teuthology.misc.health.smithi088.stderr:created 0.000000 2024-03-29T10:40:58.113 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 0 (unknown) 2024-03-29T10:40:58.113 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:40:58.113 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-03-29T10:40:58.113 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-29T10:40:58.113 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-c 2024-03-29T10:40:58.113 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:40:58.113 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:58.120+0000 7faa449f4640 1 librados: starting msgr at 2024-03-29T10:40:58.113 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:58.120+0000 7faa449f4640 1 librados: starting objecter 2024-03-29T10:40:58.113 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:58.120+0000 7faa449f4640 1 Processor -- start 2024-03-29T10:40:58.113 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:58.121+0000 7faa449f4640 1 -- start start 2024-03-29T10:40:58.113 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:58.121+0000 7faa449f4640 1 librados: setting wanted keys 2024-03-29T10:40:58.114 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:58.121+0000 7faa449f4640 1 librados: calling monclient init 2024-03-29T10:40:58.114 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:58.121+0000 7faa449f4640 10 monclient: init 2024-03-29T10:40:58.114 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:58.121+0000 7faa449f4640 10 monclient: _reopen_session rank -1 2024-03-29T10:40:58.114 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:58.121+0000 7faa449f4640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:40:58.114 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:58.121+0000 7faa449f4640 10 monclient(hunting): picked mon.noname-b con 0x7faa4006ade0 addr v1:172.21.15.196:6789/0 2024-03-29T10:40:58.114 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:58.121+0000 7faa449f4640 10 monclient(hunting): picked mon.noname-a con 0x7faa4013d830 addr v1:172.21.15.88:6789/0 2024-03-29T10:40:58.114 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:58.121+0000 7faa449f4640 10 monclient(hunting): picked mon.noname-c con 0x7faa401405a0 addr v1:172.21.15.88:6790/0 2024-03-29T10:40:58.114 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:58.122+0000 7faa449f4640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa400aae40 con 0x7faa4013d830 2024-03-29T10:40:58.114 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:58.122+0000 7faa449f4640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa401d2c30 con 0x7faa4006ade0 2024-03-29T10:40:58.114 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:58.122+0000 7faa449f4640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa400ab090 con 0x7faa401405a0 2024-03-29T10:40:58.114 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:58.122+0000 7faa449f4640 10 monclient(hunting): _renew_subs 2024-03-29T10:40:58.115 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:58.122+0000 7faa3dd9b640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7faa4013d830 0x7faa401d30c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:43328/0 (socket says 172.21.15.88:43328) 2024-03-29T10:40:58.115 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:58.122+0000 7faa3dd9b640 1 -- 172.21.15.88:0/3826571309 learned_addr learned my addr 172.21.15.88:0/3826571309 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-29T10:40:58.115 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:58.122+0000 7faa449f4640 10 monclient(hunting): authenticate will time out at 780.358052s 2024-03-29T10:40:58.115 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:58.122+0000 7faa2effd640 10 client.?.objecter ms_handle_connect 0x7faa4013d830 2024-03-29T10:40:58.115 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:58.122+0000 7faa2effd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:40:58.115 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:58.122+0000 7faa3e59c640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7faa4006ade0 0x7faa400a97e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.88:44458/0 (socket says 172.21.15.88:44458) 2024-03-29T10:40:58.115 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:58.122+0000 7faa2effd640 10 client.?.objecter ms_handle_connect 0x7faa401405a0 2024-03-29T10:40:58.115 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:58.122+0000 7faa2effd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:40:58.115 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:58.122+0000 7faa2effd640 10 client.?.objecter ms_handle_connect 0x7faa4006ade0 2024-03-29T10:40:58.115 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:58.122+0000 7faa2effd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:40:58.115 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:58.122+0000 7faa2effd640 1 -- 172.21.15.88:0/3826571309 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3587871905 0 0) 0x7faa400ab090 con 0x7faa401405a0 2024-03-29T10:40:58.115 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:58.122+0000 7faa2effd640 10 monclient(hunting): _negotiate 2024-03-29T10:40:58.115 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:58.122+0000 7faa2effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:58.116 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:58.122+0000 7faa2effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:58.116 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:58.122+0000 7faa2effd640 10 monclient(hunting): my global_id is 4238 2024-03-29T10:40:58.116 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:58.122+0000 7faa2effd640 1 -- 172.21.15.88:0/3826571309 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa20004eb0 con 0x7faa401405a0 2024-03-29T10:40:58.116 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:58.122+0000 7faa2effd640 1 -- 172.21.15.88:0/3826571309 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2224872970 0 0) 0x7faa400aae40 con 0x7faa4013d830 2024-03-29T10:40:58.116 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:58.122+0000 7faa2effd640 10 monclient(hunting): _negotiate 2024-03-29T10:40:58.116 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:58.122+0000 7faa2effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:58.116 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:58.122+0000 7faa2effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:58.116 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:58.122+0000 7faa2effd640 10 monclient(hunting): my global_id is 4245 2024-03-29T10:40:58.116 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:58.122+0000 7faa2effd640 1 -- 172.21.15.88:0/3826571309 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa400ab090 con 0x7faa4013d830 2024-03-29T10:40:58.116 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:58.122+0000 7faa2effd640 1 -- 172.21.15.88:0/3826571309 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3885189626 0 0) 0x7faa401d2c30 con 0x7faa4006ade0 2024-03-29T10:40:58.116 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:58.122+0000 7faa2effd640 10 monclient(hunting): _negotiate 2024-03-29T10:40:58.117 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:58.122+0000 7faa2effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:58.117 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:58.122+0000 7faa2effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:58.117 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:58.122+0000 7faa2effd640 10 monclient(hunting): my global_id is 4237 2024-03-29T10:40:58.117 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:58.122+0000 7faa2effd640 1 -- 172.21.15.88:0/3826571309 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa400aae40 con 0x7faa4006ade0 2024-03-29T10:40:58.117 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:58.123+0000 7faa2effd640 1 -- 172.21.15.88:0/3826571309 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2027956734 0 0) 0x7faa20004eb0 con 0x7faa401405a0 2024-03-29T10:40:58.117 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:58.123+0000 7faa2effd640 1 -- 172.21.15.88:0/3826571309 --> v1:172.21.15.88:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7faa20006680 con 0x7faa401405a0 2024-03-29T10:40:58.117 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:58.123+0000 7faa2effd640 1 -- 172.21.15.88:0/3826571309 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7faa38001a30 con 0x7faa401405a0 2024-03-29T10:40:58.117 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:58.123+0000 7faa2effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:58.117 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:58.123+0000 7faa2effd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:40:58.117 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:58.123+0000 7faa2effd640 10 monclient(hunting): dump: 2024-03-29T10:40:58.117 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:40:58.117 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:40:58.117 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:40:58.118 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:40:58.118 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:58.118 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:40:58.118 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:40:58.118 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:40:58.118 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:40:58.118 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:40:58.118 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:58.123+0000 7faa2effd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:40:58.118 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:58.123+0000 7faa2effd640 1 -- 172.21.15.88:0/3826571309 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3556617672 0 0) 0x7faa400ab090 con 0x7faa4013d830 2024-03-29T10:40:58.118 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:58.123+0000 7faa2effd640 1 -- 172.21.15.88:0/3826571309 --> v1:172.21.15.88:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7faa20004eb0 con 0x7faa4013d830 2024-03-29T10:40:58.118 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:58.123+0000 7faa2effd640 1 -- 172.21.15.88:0/3826571309 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7faa34001b40 con 0x7faa4013d830 2024-03-29T10:40:58.118 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:58.123+0000 7faa2effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:58.118 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:58.123+0000 7faa2effd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:40:58.118 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:58.123+0000 7faa2effd640 10 monclient(hunting): dump: 2024-03-29T10:40:58.119 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:40:58.119 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:40:58.119 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:40:58.119 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:40:58.119 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:58.119 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:40:58.119 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:40:58.119 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:40:58.119 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:40:58.119 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:40:58.119 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:58.123+0000 7faa2effd640 1 -- 172.21.15.88:0/3826571309 <== mon.2 v1:172.21.15.88:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2502926898 0 0) 0x7faa20006680 con 0x7faa401405a0 2024-03-29T10:40:58.119 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:58.123+0000 7faa2effd640 1 -- 172.21.15.88:0/3826571309 >> v1:172.21.15.196:6789/0 conn(0x7faa4006ade0 legacy=0x7faa400a97e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:58.119 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:58.123+0000 7faa2effd640 1 -- 172.21.15.88:0/3826571309 >> v1:172.21.15.88:6789/0 conn(0x7faa4013d830 legacy=0x7faa401d30c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:58.120 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:58.123+0000 7faa2effd640 10 monclient: _finish_hunting 0 2024-03-29T10:40:58.120 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:58.123+0000 7faa2effd640 1 monclient: found mon.c 2024-03-29T10:40:58.120 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:58.123+0000 7faa2effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:40:58.120 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:58.123+0000 7faa2effd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-03-29T10:40:58.120 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:58.123+0000 7faa2effd640 1 -- 172.21.15.88:0/3826571309 --> v1:172.21.15.88:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa400a9ef0 con 0x7faa401405a0 2024-03-29T10:40:58.120 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:58.124+0000 7faa2effd640 10 monclient: _finish_auth 0 2024-03-29T10:40:58.120 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:58.124+0000 7faa2effd640 10 monclient: _check_auth_tickets 2024-03-29T10:40:58.120 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:58.124+0000 7faa2effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:40:58.120 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:58.124+0000 7faa449f4640 5 monclient: authenticate success, global_id 4238 2024-03-29T10:40:58.120 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:58.124+0000 7faa449f4640 10 monclient: _renew_subs 2024-03-29T10:40:58.120 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:58.124+0000 7faa449f4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-03-29T10:40:58.120 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:58.124+0000 7faa449f4640 1 -- 172.21.15.88:0/3826571309 --> v1:172.21.15.88:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa400718e0 con 0x7faa401405a0 2024-03-29T10:40:58.120 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:58.124+0000 7faa449f4640 10 client.4238.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:40:58.121 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:58.124+0000 7faa449f4640 10 monclient: _renew_subs 2024-03-29T10:40:58.121 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:58.124+0000 7faa449f4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-03-29T10:40:58.121 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:58.124+0000 7faa449f4640 1 -- 172.21.15.88:0/3826571309 --> v1:172.21.15.88:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7faa401d39b0 con 0x7faa401405a0 2024-03-29T10:40:58.121 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:58.124+0000 7faa449f4640 1 librados: init done 2024-03-29T10:40:58.121 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:58.124+0000 7faa2effd640 1 -- 172.21.15.88:0/3826571309 <== mon.2 v1:172.21.15.88:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7faa38003970 con 0x7faa401405a0 2024-03-29T10:40:58.121 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:58.124+0000 7faa2effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:40:58.121 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:58.124+0000 7faa2effd640 1 -- 172.21.15.88:0/3826571309 <== mon.2 v1:172.21.15.88:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7faa38003c80 con 0x7faa401405a0 2024-03-29T10:40:58.121 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:58.124+0000 7faa2effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:58.121 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:58.124+0000 7faa2effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:40:58.121 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:58.124+0000 7faa2effd640 10 monclient: dump: 2024-03-29T10:40:58.121 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:40:58.121 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:40:58.121 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:40:58.121 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:40:58.122 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:58.122 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:40:58.122 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:40:58.122 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:40:58.122 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:40:58.122 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:40:58.122 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:58.124+0000 7faa2ffff640 4 set_mon_vals no callback set 2024-03-29T10:40:58.122 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:58.126+0000 7faa449f4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:40:58.122 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:58.126+0000 7faa2effd640 1 -- 172.21.15.88:0/3826571309 <== mon.2 v1:172.21.15.88:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4235629220 0 0) 0x7faa38019c50 con 0x7faa401405a0 2024-03-29T10:40:58.122 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:58.127+0000 7faa2effd640 1 -- 172.21.15.88:0/3826571309 <== mon.2 v1:172.21.15.88:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4257673582 0 0) 0x7faa38083b80 con 0x7faa401405a0 2024-03-29T10:40:58.122 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:58.127+0000 7faa2effd640 10 client.4238.objecter ms_dispatch 0x7faa40000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:40:58.122 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:58.127+0000 7faa2effd640 3 client.4238.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:40:58.122 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:58.127+0000 7faa2effd640 3 client.4238.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:40:58.122 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:58.127+0000 7faa2effd640 20 client.4238.objecter dump_active .. 0 homeless 2024-03-29T10:40:58.123 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:58.127+0000 7faa449f4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:40:58.123 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:58.127+0000 7faa449f4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-03-29T10:40:58.123 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:58.127+0000 7faa449f4640 1 -- 172.21.15.88:0/3826571309 --> v1:172.21.15.88:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa24005080 con 0x7faa401405a0 2024-03-29T10:40:58.123 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:58.127+0000 7faa3e59c640 10 monclient: get_auth_request con 0x7faa20076980 auth_method 0 2024-03-29T10:40:58.123 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:58.128+0000 7faa2effd640 10 client.4238.objecter ms_handle_connect 0x7faa20076980 2024-03-29T10:40:58.123 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:58.135+0000 7faa2effd640 1 -- 172.21.15.88:0/3826571309 <== mon.2 v1:172.21.15.88:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7faa3804f460 con 0x7faa401405a0 2024-03-29T10:40:58.123 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:58.135+0000 7faa2effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:40:58.123 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:58.135+0000 7faa2effd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:40:58.489 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:58.505+0000 7faa449f4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:40:58.489 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:58.505+0000 7faa449f4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:40:58.489 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:58.505+0000 7faa449f4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-03-29T10:40:58.490 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:58.505+0000 7faa449f4640 1 -- 172.21.15.88:0/3826571309 --> v1:172.21.15.88:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7faa24005330 con 0x7faa401405a0 2024-03-29T10:40:58.490 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:58.506+0000 7faa2effd640 1 -- 172.21.15.88:0/3826571309 <== mon.2 v1:172.21.15.88:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 344506374) 0x7faa38051390 con 0x7faa401405a0 2024-03-29T10:40:58.490 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:58.506+0000 7faa2effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:40:58.490 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:58.506+0000 7faa2effd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:40:58.491 INFO:teuthology.misc.health.smithi088.stdout: 2024-03-29T10:40:58.491 INFO:teuthology.misc.health.smithi088.stdout:{"epoch":10,"fsid":"869f5188-f92b-42dd-a46d-8c646e0388a9","created":"2024-03-29T10:40:40.285087+0000","modified":"2024-03-29T10:40:42.536230+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:40:42.536230+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a26c0c1c-8de0-438c-a3f6-3df10ea61574","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5de24915-8dec-49b6-8b16-f3170197c591","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ac054af3-d589-4565-9644-e6af75975c53","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3ec4f045-c714-4a22-bd1a-e07330b45ab3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"cddb9867-8e30-4951-9b13-3d1e78b9c45a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dc6d9911-785d-432f-8b53-56a1ccab302f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"77c88437-b587-496e-8b4d-1824ea8b5305","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ce963d96-5a8e-44c3-99de-c0dbebe234ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:40:58.494 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:58.510+0000 7faa449f4640 10 librados: watch_flush enter 2024-03-29T10:40:58.494 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:58.510+0000 7faa449f4640 10 librados: watch_flush exit 2024-03-29T10:40:58.494 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:58.510+0000 7faa449f4640 20 client.4238.objecter shutdown clearing up homeless session... 2024-03-29T10:40:58.494 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:58.510+0000 7faa449f4640 10 client.4238.objecter successfully canceled tick 2024-03-29T10:40:58.494 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:58.510+0000 7faa449f4640 1 -- 172.21.15.88:0/3826571309 >> v1:172.21.15.88:6816/1752861349 conn(0x7faa20076980 legacy=0x7faa20078e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:58.494 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:58.510+0000 7faa449f4640 10 monclient: shutdown 2024-03-29T10:40:58.494 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:58.510+0000 7faa449f4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:40:58.494 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:58.510+0000 7faa449f4640 1 -- 172.21.15.88:0/3826571309 >> v1:172.21.15.88:6790/0 conn(0x7faa401405a0 legacy=0x7faa401349e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:58.495 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:58.510+0000 7faa449f4640 1 -- 172.21.15.88:0/3826571309 shutdown_connections 2024-03-29T10:40:58.495 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:58.510+0000 7faa449f4640 1 -- 172.21.15.88:0/3826571309 >> 172.21.15.88:0/3826571309 conn(0x7faa400b4930 msgr2=0x7faa4013cef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:40:58.495 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:58.511+0000 7faa449f4640 1 -- 172.21.15.88:0/3826571309 shutdown_connections 2024-03-29T10:40:58.495 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:58.511+0000 7faa449f4640 1 -- 172.21.15.88:0/3826571309 wait complete. 2024-03-29T10:40:58.495 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:40:58.511+0000 7faa449f4640 1 librados: shutdown 2024-03-29T10:40:58.509 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:41:05.317 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:41:05.465 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:05.480+0000 7f6d6dc86640 10 monclient: get_monmap_and_config 2024-03-29T10:41:05.465 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:05.480+0000 7f6d6dc86640 10 monclient: build_initial_monmap 2024-03-29T10:41:05.465 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:05.480+0000 7f6d6dc86640 10 monclient: monmap: 2024-03-29T10:41:05.465 INFO:teuthology.misc.health.smithi088.stderr:epoch 0 2024-03-29T10:41:05.465 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:41:05.465 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:41:05.481550+0000 2024-03-29T10:41:05.465 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:41:05.481550+0000 2024-03-29T10:41:05.465 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 0 (unknown) 2024-03-29T10:41:05.465 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:41:05.465 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-03-29T10:41:05.466 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-29T10:41:05.466 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-b 2024-03-29T10:41:05.466 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:41:05.466 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:05.481+0000 7f6d6dc86640 1 Processor -- start 2024-03-29T10:41:05.466 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:05.481+0000 7f6d6dc86640 1 -- start start 2024-03-29T10:41:05.466 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:05.481+0000 7f6d6dc86640 10 monclient: init 2024-03-29T10:41:05.470 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:05.482+0000 7f6d6dc86640 10 monclient: _reopen_session rank -1 2024-03-29T10:41:05.470 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:05.482+0000 7f6d6dc86640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:41:05.470 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:05.482+0000 7f6d6dc86640 10 monclient(hunting): picked mon.noname-b con 0x7f6d680b4100 addr v1:172.21.15.88:6790/0 2024-03-29T10:41:05.470 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:05.482+0000 7f6d6dc86640 10 monclient(hunting): picked mon.noname-a con 0x7f6d6813ee90 addr v1:172.21.15.88:6789/0 2024-03-29T10:41:05.471 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:05.484+0000 7f6d6dc86640 10 monclient(hunting): picked mon.noname-c con 0x7f6d680af900 addr v1:172.21.15.196:6789/0 2024-03-29T10:41:05.471 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:05.484+0000 7f6d6dc86640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d680afd10 con 0x7f6d6813ee90 2024-03-29T10:41:05.471 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:05.484+0000 7f6d6dc86640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d680af6f0 con 0x7f6d680af900 2024-03-29T10:41:05.471 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:05.484+0000 7f6d6dc86640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d680aaf60 con 0x7f6d680b4100 2024-03-29T10:41:05.471 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:05.484+0000 7f6d6dc86640 10 monclient(hunting): _renew_subs 2024-03-29T10:41:05.471 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:05.484+0000 7f6d6dc86640 10 monclient(hunting): authenticate will time out at 787.720482s 2024-03-29T10:41:05.471 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:05.484+0000 7f6d67fff640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f6d680af900 0x7f6d68149b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.88:41586/0 (socket says 172.21.15.88:41586) 2024-03-29T10:41:05.471 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:05.484+0000 7f6d67fff640 1 -- 172.21.15.88:0/1526857384 learned_addr learned my addr 172.21.15.88:0/1526857384 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-29T10:41:05.471 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:05.486+0000 7f6d667fc640 1 -- 172.21.15.88:0/1526857384 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 792132200 0 0) 0x7f6d680af6f0 con 0x7f6d680af900 2024-03-29T10:41:05.471 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:05.486+0000 7f6d667fc640 10 monclient(hunting): _negotiate 2024-03-29T10:41:05.471 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:05.486+0000 7f6d667fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:41:05.471 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:05.486+0000 7f6d667fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:41:05.471 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:05.486+0000 7f6d667fc640 10 monclient(hunting): my global_id is 4240 2024-03-29T10:41:05.471 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:05.486+0000 7f6d667fc640 1 -- 172.21.15.88:0/1526857384 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6d50003520 con 0x7f6d680af900 2024-03-29T10:41:05.472 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:05.486+0000 7f6d667fc640 1 -- 172.21.15.88:0/1526857384 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3559414919 0 0) 0x7f6d50003520 con 0x7f6d680af900 2024-03-29T10:41:05.472 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:05.486+0000 7f6d667fc640 1 -- 172.21.15.88:0/1526857384 >> v1:172.21.15.88:6790/0 conn(0x7f6d680b4100 legacy=0x7f6d680a9850 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:41:05.472 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:05.488+0000 7f6d667fc640 1 -- 172.21.15.88:0/1526857384 >> v1:172.21.15.88:6789/0 conn(0x7f6d6813ee90 legacy=0x7f6d6813b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:05.472 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:05.488+0000 7f6d667fc640 10 monclient: _finish_hunting 0 2024-03-29T10:41:05.472 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:05.488+0000 7f6d667fc640 1 monclient: found mon.noname-c 2024-03-29T10:41:05.472 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:05.488+0000 7f6d667fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:41:05.472 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:05.488+0000 7f6d667fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.196:6789/0 2024-03-29T10:41:05.473 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:05.488+0000 7f6d667fc640 1 -- 172.21.15.88:0/1526857384 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d6813e970 con 0x7f6d680af900 2024-03-29T10:41:05.473 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:05.488+0000 7f6d667fc640 10 monclient: _finish_auth 0 2024-03-29T10:41:05.473 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:05.488+0000 7f6d667fc640 10 monclient: _check_auth_tickets 2024-03-29T10:41:05.473 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:05.488+0000 7f6d667fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:41:05.473 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:05.488+0000 7f6d667fc640 1 -- 172.21.15.88:0/1526857384 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f6d60001a10 con 0x7f6d680af900 2024-03-29T10:41:05.473 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:05.488+0000 7f6d667fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:05.473 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:05.488+0000 7f6d667fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:41:05.473 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:05.488+0000 7f6d667fc640 10 monclient: dump: 2024-03-29T10:41:05.473 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:41:05.473 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:41:05.473 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:41:05.473 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:41:05.473 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:05.474 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:41:05.474 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:41:05.474 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:41:05.474 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:41:05.474 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:41:05.474 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:05.489+0000 7f6d667fc640 1 -- 172.21.15.88:0/1526857384 <== mon.1 v1:172.21.15.196:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6d60001ca0 con 0x7f6d680af900 2024-03-29T10:41:05.474 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:05.489+0000 7f6d667fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:41:05.474 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:05.489+0000 7f6d667fc640 1 -- 172.21.15.88:0/1526857384 <== mon.1 v1:172.21.15.196:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f6d60002030 con 0x7f6d680af900 2024-03-29T10:41:05.474 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:05.489+0000 7f6d667fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:05.474 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:05.489+0000 7f6d667fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:41:05.474 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:05.489+0000 7f6d667fc640 10 monclient: dump: 2024-03-29T10:41:05.474 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:41:05.474 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:41:05.474 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:41:05.475 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:41:05.475 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:05.475 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:41:05.475 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:41:05.475 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:41:05.475 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:41:05.475 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:41:05.475 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:05.489+0000 7f6d6dc86640 5 monclient: authenticate success, global_id 4240 2024-03-29T10:41:05.475 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:05.489+0000 7f6d6dc86640 10 monclient: get_monmap_and_config success 2024-03-29T10:41:05.475 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:05.489+0000 7f6d6dc86640 4 set_mon_vals no callback set 2024-03-29T10:41:05.477 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:05.489+0000 7f6d6dc86640 10 monclient: shutdown 2024-03-29T10:41:05.477 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:05.489+0000 7f6d6dc86640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:41:05.478 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:05.489+0000 7f6d6dc86640 1 -- 172.21.15.88:0/1526857384 >> v1:172.21.15.196:6789/0 conn(0x7f6d680af900 legacy=0x7f6d68149b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:05.478 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:05.490+0000 7f6d6dc86640 1 -- 172.21.15.88:0/1526857384 shutdown_connections 2024-03-29T10:41:05.478 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:05.490+0000 7f6d6dc86640 1 -- 172.21.15.88:0/1526857384 >> 172.21.15.88:0/1526857384 conn(0x7f6d680b35e0 msgr2=0x7f6d680b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:41:05.478 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:05.490+0000 7f6d6dc86640 1 -- 172.21.15.88:0/1526857384 shutdown_connections 2024-03-29T10:41:05.478 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:05.491+0000 7f6d6dc86640 1 -- 172.21.15.88:0/1526857384 wait complete. 2024-03-29T10:41:05.478 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:05.492+0000 7f6d6dc86640 10 monclient: build_initial_monmap 2024-03-29T10:41:05.478 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:05.492+0000 7f6d6dc86640 10 monclient: monmap: 2024-03-29T10:41:05.478 INFO:teuthology.misc.health.smithi088.stderr:epoch 0 2024-03-29T10:41:05.478 INFO:teuthology.misc.health.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:41:05.478 INFO:teuthology.misc.health.smithi088.stderr:last_changed 0.000000 2024-03-29T10:41:05.479 INFO:teuthology.misc.health.smithi088.stderr:created 0.000000 2024-03-29T10:41:05.479 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 0 (unknown) 2024-03-29T10:41:05.479 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:41:05.479 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-03-29T10:41:05.479 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-29T10:41:05.479 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-c 2024-03-29T10:41:05.479 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:41:05.479 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:05.492+0000 7f6d6dc86640 1 librados: starting msgr at 2024-03-29T10:41:05.479 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:05.492+0000 7f6d6dc86640 1 librados: starting objecter 2024-03-29T10:41:05.479 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:05.492+0000 7f6d6dc86640 1 Processor -- start 2024-03-29T10:41:05.479 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:05.493+0000 7f6d6dc86640 1 -- start start 2024-03-29T10:41:05.480 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:05.493+0000 7f6d6dc86640 1 librados: setting wanted keys 2024-03-29T10:41:05.480 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:05.493+0000 7f6d6dc86640 1 librados: calling monclient init 2024-03-29T10:41:05.480 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:05.493+0000 7f6d6dc86640 10 monclient: init 2024-03-29T10:41:05.480 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:05.494+0000 7f6d6dc86640 10 monclient: _reopen_session rank -1 2024-03-29T10:41:05.480 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:05.494+0000 7f6d6dc86640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:41:05.480 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:05.494+0000 7f6d6dc86640 10 monclient(hunting): picked mon.noname-b con 0x7f6d680af900 addr v1:172.21.15.196:6789/0 2024-03-29T10:41:05.480 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:05.494+0000 7f6d6dc86640 10 monclient(hunting): picked mon.noname-a con 0x7f6d680b4100 addr v1:172.21.15.88:6789/0 2024-03-29T10:41:05.482 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:05.496+0000 7f6d6dc86640 10 monclient(hunting): picked mon.noname-c con 0x7f6d6813ee90 addr v1:172.21.15.88:6790/0 2024-03-29T10:41:05.482 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:05.496+0000 7f6d6dc86640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d681d70d0 con 0x7f6d680b4100 2024-03-29T10:41:05.482 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:05.496+0000 7f6d6dc86640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d68072380 con 0x7f6d680af900 2024-03-29T10:41:05.482 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:05.496+0000 7f6d6dc86640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d6813d510 con 0x7f6d6813ee90 2024-03-29T10:41:05.482 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:05.496+0000 7f6d6dc86640 10 monclient(hunting): _renew_subs 2024-03-29T10:41:05.482 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:05.496+0000 7f6d6dc86640 10 monclient(hunting): authenticate will time out at 787.732499s 2024-03-29T10:41:05.482 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:05.494+0000 7f6d677fe640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f6d680af900 0x7f6d68070800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.88:41778/0 (socket says 172.21.15.88:41778) 2024-03-29T10:41:05.482 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:05.498+0000 7f6d677fe640 1 -- 172.21.15.88:0/2778732858 learned_addr learned my addr 172.21.15.88:0/2778732858 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-29T10:41:05.488 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:05.499+0000 7f6d4ffff640 10 client.?.objecter ms_handle_connect 0x7f6d680af900 2024-03-29T10:41:05.488 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:05.499+0000 7f6d4ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:41:05.488 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:05.499+0000 7f6d4ffff640 10 client.?.objecter ms_handle_connect 0x7f6d680b4100 2024-03-29T10:41:05.488 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:05.499+0000 7f6d4ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:41:05.488 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:05.499+0000 7f6d4ffff640 1 -- 172.21.15.88:0/2778732858 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1561812195 0 0) 0x7f6d68072380 con 0x7f6d680af900 2024-03-29T10:41:05.488 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:05.499+0000 7f6d4ffff640 10 monclient(hunting): _negotiate 2024-03-29T10:41:05.488 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:05.499+0000 7f6d4ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:41:05.488 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:05.499+0000 7f6d4ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:41:05.488 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:05.499+0000 7f6d4ffff640 10 monclient(hunting): my global_id is 4243 2024-03-29T10:41:05.488 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:05.500+0000 7f6d4ffff640 1 -- 172.21.15.88:0/2778732858 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6d54004eb0 con 0x7f6d680af900 2024-03-29T10:41:05.488 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:05.500+0000 7f6d4ffff640 10 client.?.objecter ms_handle_connect 0x7f6d6813ee90 2024-03-29T10:41:05.489 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:05.500+0000 7f6d4ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:41:05.489 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:05.500+0000 7f6d4ffff640 1 -- 172.21.15.88:0/2778732858 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 77198546 0 0) 0x7f6d681d70d0 con 0x7f6d680b4100 2024-03-29T10:41:05.489 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:05.500+0000 7f6d4ffff640 10 monclient(hunting): _negotiate 2024-03-29T10:41:05.489 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:05.500+0000 7f6d4ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:41:05.489 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:05.500+0000 7f6d4ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:41:05.489 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:05.500+0000 7f6d4ffff640 10 monclient(hunting): my global_id is 4251 2024-03-29T10:41:05.489 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:05.500+0000 7f6d4ffff640 1 -- 172.21.15.88:0/2778732858 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6d68072380 con 0x7f6d680b4100 2024-03-29T10:41:05.489 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:05.500+0000 7f6d4ffff640 1 -- 172.21.15.88:0/2778732858 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1322907379 0 0) 0x7f6d6813d510 con 0x7f6d6813ee90 2024-03-29T10:41:05.489 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:05.500+0000 7f6d4ffff640 10 monclient(hunting): _negotiate 2024-03-29T10:41:05.489 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:05.500+0000 7f6d4ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:41:05.489 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:05.500+0000 7f6d4ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:41:05.489 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:05.500+0000 7f6d4ffff640 10 monclient(hunting): my global_id is 4241 2024-03-29T10:41:05.489 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:05.500+0000 7f6d4ffff640 1 -- 172.21.15.88:0/2778732858 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6d681d70d0 con 0x7f6d6813ee90 2024-03-29T10:41:05.489 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:05.501+0000 7f6d4ffff640 1 -- 172.21.15.88:0/2778732858 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 395619062 0 0) 0x7f6d54004eb0 con 0x7f6d680af900 2024-03-29T10:41:05.490 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:05.501+0000 7f6d4ffff640 1 -- 172.21.15.88:0/2778732858 --> v1:172.21.15.196:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6d6813d510 con 0x7f6d680af900 2024-03-29T10:41:05.490 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:05.501+0000 7f6d4ffff640 1 -- 172.21.15.88:0/2778732858 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2197755745 0 0) 0x7f6d68072380 con 0x7f6d680b4100 2024-03-29T10:41:05.490 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:05.501+0000 7f6d4ffff640 1 -- 172.21.15.88:0/2778732858 --> v1:172.21.15.88:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6d54004eb0 con 0x7f6d680b4100 2024-03-29T10:41:05.490 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:05.501+0000 7f6d4ffff640 1 -- 172.21.15.88:0/2778732858 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f6d58004080 con 0x7f6d680af900 2024-03-29T10:41:05.490 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:05.501+0000 7f6d4ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:05.490 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:05.501+0000 7f6d4ffff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:41:05.490 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:05.501+0000 7f6d4ffff640 10 monclient(hunting): dump: 2024-03-29T10:41:05.490 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:41:05.490 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:41:05.490 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:41:05.490 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:41:05.490 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:05.490 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:41:05.490 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:41:05.491 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:41:05.491 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:41:05.491 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:41:05.491 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:05.501+0000 7f6d4ffff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:41:05.491 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:05.501+0000 7f6d4ffff640 1 -- 172.21.15.88:0/2778732858 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f6d5c0016c0 con 0x7f6d680b4100 2024-03-29T10:41:05.491 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:05.501+0000 7f6d4ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:05.491 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:05.501+0000 7f6d4ffff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:41:05.491 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:05.501+0000 7f6d4ffff640 10 monclient(hunting): dump: 2024-03-29T10:41:05.491 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:41:05.491 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:41:05.491 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:41:05.491 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:41:05.491 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:05.491 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:41:05.492 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:41:05.492 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:41:05.492 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:41:05.492 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:41:05.492 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:05.501+0000 7f6d4ffff640 1 -- 172.21.15.88:0/2778732858 <== mon.0 v1:172.21.15.88:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2186278938 0 0) 0x7f6d54004eb0 con 0x7f6d680b4100 2024-03-29T10:41:05.492 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:05.501+0000 7f6d4ffff640 1 -- 172.21.15.88:0/2778732858 >> v1:172.21.15.88:6790/0 conn(0x7f6d6813ee90 legacy=0x7f6d681d69c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:05.492 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:05.502+0000 7f6d4ffff640 1 -- 172.21.15.88:0/2778732858 >> v1:172.21.15.196:6789/0 conn(0x7f6d680af900 legacy=0x7f6d68070800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:05.492 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:05.502+0000 7f6d4ffff640 10 monclient: _finish_hunting 0 2024-03-29T10:41:05.492 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:05.502+0000 7f6d4ffff640 1 monclient: found mon.a 2024-03-29T10:41:05.492 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:05.502+0000 7f6d4ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:41:05.492 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:05.502+0000 7f6d4ffff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-03-29T10:41:05.492 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:05.502+0000 7f6d4ffff640 1 -- 172.21.15.88:0/2778732858 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d6813d7f0 con 0x7f6d680b4100 2024-03-29T10:41:05.492 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:05.502+0000 7f6d4ffff640 10 monclient: _finish_auth 0 2024-03-29T10:41:05.493 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:05.502+0000 7f6d4ffff640 10 monclient: _check_auth_tickets 2024-03-29T10:41:05.493 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:05.502+0000 7f6d4ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:41:05.493 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:05.502+0000 7f6d6dc86640 5 monclient: authenticate success, global_id 4251 2024-03-29T10:41:05.493 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:05.502+0000 7f6d6dc86640 10 monclient: _renew_subs 2024-03-29T10:41:05.493 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:05.502+0000 7f6d6dc86640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-03-29T10:41:05.493 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:05.502+0000 7f6d6dc86640 1 -- 172.21.15.88:0/2778732858 --> v1:172.21.15.88:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d68135ad0 con 0x7f6d680b4100 2024-03-29T10:41:05.493 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:05.502+0000 7f6d6dc86640 10 client.4251.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:41:05.493 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:05.502+0000 7f6d6dc86640 10 monclient: _renew_subs 2024-03-29T10:41:05.493 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:05.502+0000 7f6d6dc86640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-03-29T10:41:05.493 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:05.502+0000 7f6d6dc86640 1 -- 172.21.15.88:0/2778732858 --> v1:172.21.15.88:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d68070270 con 0x7f6d680b4100 2024-03-29T10:41:05.493 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:05.502+0000 7f6d6dc86640 1 librados: init done 2024-03-29T10:41:05.493 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:05.504+0000 7f6d6dc86640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:41:05.493 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:05.504+0000 7f6d6dc86640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:41:05.493 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:05.504+0000 7f6d6dc86640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-03-29T10:41:05.493 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:05.504+0000 7f6d6dc86640 1 -- 172.21.15.88:0/2778732858 --> v1:172.21.15.88:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d68075f10 con 0x7f6d680b4100 2024-03-29T10:41:05.494 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:05.504+0000 7f6d4ffff640 1 -- 172.21.15.88:0/2778732858 <== mon.0 v1:172.21.15.88:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6d5c001bd0 con 0x7f6d680b4100 2024-03-29T10:41:05.494 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:05.504+0000 7f6d4ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:41:05.495 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:05.504+0000 7f6d65ffb640 4 set_mon_vals no callback set 2024-03-29T10:41:05.495 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:05.505+0000 7f6d4ffff640 1 -- 172.21.15.88:0/2778732858 <== mon.0 v1:172.21.15.88:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f6d5c0037b0 con 0x7f6d680b4100 2024-03-29T10:41:05.495 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:05.505+0000 7f6d4ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:05.495 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:05.505+0000 7f6d4ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:41:05.495 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:05.505+0000 7f6d4ffff640 10 monclient: dump: 2024-03-29T10:41:05.496 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:41:05.496 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:41:05.496 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:41:05.496 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:41:05.496 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:05.496 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:41:05.496 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:41:05.496 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:41:05.496 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:41:05.496 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:41:05.496 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:05.506+0000 7f6d4ffff640 1 -- 172.21.15.88:0/2778732858 <== mon.0 v1:172.21.15.88:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4235629220 0 0) 0x7f6d5c019780 con 0x7f6d680b4100 2024-03-29T10:41:05.496 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:05.506+0000 7f6d4ffff640 1 -- 172.21.15.88:0/2778732858 <== mon.0 v1:172.21.15.88:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4257673582 0 0) 0x7f6d5c083640 con 0x7f6d680b4100 2024-03-29T10:41:05.496 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:05.506+0000 7f6d677fe640 10 monclient: get_auth_request con 0x7f6d54072220 auth_method 0 2024-03-29T10:41:05.496 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:05.507+0000 7f6d4ffff640 10 client.4251.objecter ms_dispatch 0x7f6d68000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:41:05.497 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:05.507+0000 7f6d4ffff640 3 client.4251.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:41:05.497 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:05.507+0000 7f6d4ffff640 3 client.4251.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:41:05.497 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:05.507+0000 7f6d4ffff640 20 client.4251.objecter dump_active .. 0 homeless 2024-03-29T10:41:05.497 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:05.507+0000 7f6d4ffff640 10 client.4251.objecter ms_handle_connect 0x7f6d54072220 2024-03-29T10:41:05.502 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:05.519+0000 7f6d4ffff640 1 -- 172.21.15.88:0/2778732858 <== mon.0 v1:172.21.15.88:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6d5c04ef20 con 0x7f6d680b4100 2024-03-29T10:41:05.502 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:05.519+0000 7f6d4ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:41:05.502 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:05.519+0000 7f6d4ffff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:41:05.826 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:05.842+0000 7f6d6dc86640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:41:05.826 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:05.842+0000 7f6d6dc86640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:41:05.826 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:05.842+0000 7f6d6dc86640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-03-29T10:41:05.826 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:05.842+0000 7f6d6dc86640 1 -- 172.21.15.88:0/2778732858 --> v1:172.21.15.88:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6d6813d980 con 0x7f6d680b4100 2024-03-29T10:41:05.827 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:05.843+0000 7f6d4ffff640 1 -- 172.21.15.88:0/2778732858 <== mon.0 v1:172.21.15.88:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 344506374) 0x7f6d5c050e50 con 0x7f6d680b4100 2024-03-29T10:41:05.827 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:05.843+0000 7f6d4ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:41:05.827 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:05.843+0000 7f6d4ffff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:41:05.828 INFO:teuthology.misc.health.smithi088.stdout: 2024-03-29T10:41:05.828 INFO:teuthology.misc.health.smithi088.stdout:{"epoch":10,"fsid":"869f5188-f92b-42dd-a46d-8c646e0388a9","created":"2024-03-29T10:40:40.285087+0000","modified":"2024-03-29T10:40:42.536230+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:40:42.536230+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a26c0c1c-8de0-438c-a3f6-3df10ea61574","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5de24915-8dec-49b6-8b16-f3170197c591","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ac054af3-d589-4565-9644-e6af75975c53","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3ec4f045-c714-4a22-bd1a-e07330b45ab3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"cddb9867-8e30-4951-9b13-3d1e78b9c45a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dc6d9911-785d-432f-8b53-56a1ccab302f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"77c88437-b587-496e-8b4d-1824ea8b5305","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ce963d96-5a8e-44c3-99de-c0dbebe234ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:41:05.832 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:05.847+0000 7f6d6dc86640 10 librados: watch_flush enter 2024-03-29T10:41:05.832 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:05.847+0000 7f6d6dc86640 10 librados: watch_flush exit 2024-03-29T10:41:05.832 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:05.847+0000 7f6d6dc86640 20 client.4251.objecter shutdown clearing up homeless session... 2024-03-29T10:41:05.832 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:05.847+0000 7f6d6dc86640 10 client.4251.objecter successfully canceled tick 2024-03-29T10:41:05.832 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:05.847+0000 7f6d6dc86640 1 -- 172.21.15.88:0/2778732858 >> v1:172.21.15.88:6816/1752861349 conn(0x7f6d54072220 legacy=0x7f6d540746e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:05.832 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:05.847+0000 7f6d6dc86640 10 monclient: shutdown 2024-03-29T10:41:05.832 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:05.847+0000 7f6d6dc86640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:41:05.832 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:05.847+0000 7f6d6dc86640 1 -- 172.21.15.88:0/2778732858 >> v1:172.21.15.88:6789/0 conn(0x7f6d680b4100 legacy=0x7f6d681d62b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:05.832 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:05.848+0000 7f6d6dc86640 1 -- 172.21.15.88:0/2778732858 shutdown_connections 2024-03-29T10:41:05.832 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:05.848+0000 7f6d6dc86640 1 -- 172.21.15.88:0/2778732858 >> 172.21.15.88:0/2778732858 conn(0x7f6d680b35e0 msgr2=0x7f6d680b07c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:41:05.832 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:05.848+0000 7f6d6dc86640 1 -- 172.21.15.88:0/2778732858 shutdown_connections 2024-03-29T10:41:05.833 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:05.848+0000 7f6d6dc86640 1 -- 172.21.15.88:0/2778732858 wait complete. 2024-03-29T10:41:05.833 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:05.849+0000 7f6d6dc86640 1 librados: shutdown 2024-03-29T10:41:05.850 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:41:12.658 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:41:12.803 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:12.819+0000 7fdab759e640 10 monclient: get_monmap_and_config 2024-03-29T10:41:12.804 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:12.819+0000 7fdab759e640 10 monclient: build_initial_monmap 2024-03-29T10:41:12.804 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:12.819+0000 7fdab759e640 10 monclient: monmap: 2024-03-29T10:41:12.804 INFO:teuthology.misc.health.smithi088.stderr:epoch 0 2024-03-29T10:41:12.804 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:41:12.804 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:41:12.820629+0000 2024-03-29T10:41:12.804 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:41:12.820629+0000 2024-03-29T10:41:12.804 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 0 (unknown) 2024-03-29T10:41:12.804 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:41:12.804 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-03-29T10:41:12.804 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-29T10:41:12.804 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-b 2024-03-29T10:41:12.805 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:41:12.805 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:12.820+0000 7fdab759e640 1 Processor -- start 2024-03-29T10:41:12.805 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:12.820+0000 7fdab759e640 1 -- start start 2024-03-29T10:41:12.805 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:12.820+0000 7fdab759e640 10 monclient: init 2024-03-29T10:41:12.807 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:12.821+0000 7fdab759e640 10 monclient: _reopen_session rank -1 2024-03-29T10:41:12.807 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:12.821+0000 7fdab759e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:41:12.807 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:12.821+0000 7fdab759e640 10 monclient(hunting): picked mon.noname-a con 0x7fdab00345e0 addr v1:172.21.15.88:6789/0 2024-03-29T10:41:12.807 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:12.821+0000 7fdab759e640 10 monclient(hunting): picked mon.noname-c con 0x7fdab00349c0 addr v1:172.21.15.196:6789/0 2024-03-29T10:41:12.807 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:12.821+0000 7fdab759e640 10 monclient(hunting): picked mon.noname-b con 0x7fdab002b570 addr v1:172.21.15.88:6790/0 2024-03-29T10:41:12.807 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:12.821+0000 7fdab759e640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdab0032ef0 con 0x7fdab00345e0 2024-03-29T10:41:12.807 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:12.821+0000 7fdab759e640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdab00be170 con 0x7fdab00349c0 2024-03-29T10:41:12.808 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:12.821+0000 7fdab759e640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdab002c720 con 0x7fdab002b570 2024-03-29T10:41:12.808 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:12.821+0000 7fdab759e640 10 monclient(hunting): _renew_subs 2024-03-29T10:41:12.808 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:12.821+0000 7fdab759e640 10 monclient(hunting): authenticate will time out at 795.057682s 2024-03-29T10:41:12.810 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:12.824+0000 7fdab5d9b640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7fdab00349c0 0x7fdab00c7260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.88:33264/0 (socket says 172.21.15.88:33264) 2024-03-29T10:41:12.810 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:12.824+0000 7fdab5d9b640 1 -- 172.21.15.88:0/1684579306 learned_addr learned my addr 172.21.15.88:0/1684579306 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-29T10:41:12.810 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:12.825+0000 7fdab559a640 1 -- 172.21.15.88:0/1684579306 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 735265117 0 0) 0x7fdab002c720 con 0x7fdab002b570 2024-03-29T10:41:12.810 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:12.825+0000 7fdab559a640 10 monclient(hunting): _negotiate 2024-03-29T10:41:12.810 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:12.825+0000 7fdab559a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:41:12.810 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:12.825+0000 7fdab559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:41:12.810 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:12.825+0000 7fdab559a640 10 monclient(hunting): my global_id is 4244 2024-03-29T10:41:12.810 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:12.825+0000 7fdab559a640 1 -- 172.21.15.88:0/1684579306 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdaa4003520 con 0x7fdab002b570 2024-03-29T10:41:12.810 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:12.825+0000 7fdab559a640 1 -- 172.21.15.88:0/1684579306 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2689786909 0 0) 0x7fdab00be170 con 0x7fdab00349c0 2024-03-29T10:41:12.810 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:12.825+0000 7fdab559a640 10 monclient(hunting): _negotiate 2024-03-29T10:41:12.810 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:12.825+0000 7fdab559a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:41:12.811 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:12.825+0000 7fdab559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:41:12.811 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:12.825+0000 7fdab559a640 10 monclient(hunting): my global_id is 4246 2024-03-29T10:41:12.811 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:12.825+0000 7fdab559a640 1 -- 172.21.15.88:0/1684579306 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdab002c720 con 0x7fdab00349c0 2024-03-29T10:41:12.811 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:12.825+0000 7fdab559a640 1 -- 172.21.15.88:0/1684579306 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1457746620 0 0) 0x7fdaa4003520 con 0x7fdab002b570 2024-03-29T10:41:12.811 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:12.825+0000 7fdab559a640 1 -- 172.21.15.88:0/1684579306 >> v1:172.21.15.196:6789/0 conn(0x7fdab00349c0 legacy=0x7fdab00c7260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:12.811 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:12.825+0000 7fdab559a640 1 -- 172.21.15.88:0/1684579306 >> v1:172.21.15.88:6789/0 conn(0x7fdab00345e0 legacy=0x7fdab00bda60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:12.811 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:12.825+0000 7fdab559a640 10 monclient: _finish_hunting 0 2024-03-29T10:41:12.811 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:12.825+0000 7fdab559a640 1 monclient: found mon.noname-b 2024-03-29T10:41:12.811 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:12.825+0000 7fdab559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:41:12.811 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:12.825+0000 7fdab559a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.88:6790/0 2024-03-29T10:41:12.811 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:12.825+0000 7fdab559a640 1 -- 172.21.15.88:0/1684579306 --> v1:172.21.15.88:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdab002c990 con 0x7fdab002b570 2024-03-29T10:41:12.811 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:12.825+0000 7fdab559a640 10 monclient: _finish_auth 0 2024-03-29T10:41:12.811 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:12.825+0000 7fdab559a640 10 monclient: _check_auth_tickets 2024-03-29T10:41:12.811 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:12.825+0000 7fdab559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:41:12.812 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:12.825+0000 7fdab759e640 5 monclient: authenticate success, global_id 4244 2024-03-29T10:41:12.812 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:12.825+0000 7fdab759e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:41:12.812 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:12.825+0000 7fdab559a640 1 -- 172.21.15.88:0/1684579306 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7fdab804f730 con 0x7fdab002b570 2024-03-29T10:41:12.812 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:12.825+0000 7fdab559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:12.812 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:12.825+0000 7fdab559a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:41:12.812 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:12.825+0000 7fdab559a640 10 monclient: dump: 2024-03-29T10:41:12.812 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:41:12.812 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:41:12.812 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:41:12.812 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:41:12.812 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:12.812 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:41:12.812 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:41:12.812 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:41:12.813 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:41:12.813 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:41:12.813 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:12.825+0000 7fdab759e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:41:12.813 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:12.825+0000 7fdab559a640 1 -- 172.21.15.88:0/1684579306 <== mon.2 v1:172.21.15.88:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdab80008d0 con 0x7fdab002b570 2024-03-29T10:41:12.813 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:12.825+0000 7fdab559a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:41:12.813 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:12.825+0000 7fdab759e640 10 monclient: get_monmap_and_config success 2024-03-29T10:41:12.813 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:12.825+0000 7fdab759e640 4 set_mon_vals no callback set 2024-03-29T10:41:12.813 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:12.826+0000 7fdab559a640 1 -- 172.21.15.88:0/1684579306 <== mon.2 v1:172.21.15.88:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7fdab80758b0 con 0x7fdab002b570 2024-03-29T10:41:12.813 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:12.826+0000 7fdab559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:12.813 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:12.826+0000 7fdab559a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:41:12.813 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:12.826+0000 7fdab559a640 10 monclient: dump: 2024-03-29T10:41:12.813 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:41:12.813 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:41:12.813 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:41:12.814 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:41:12.814 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:12.814 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:41:12.814 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:41:12.814 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:41:12.814 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:41:12.814 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:41:12.816 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:12.826+0000 7fdab759e640 10 monclient: shutdown 2024-03-29T10:41:12.816 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:12.826+0000 7fdab759e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:41:12.816 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:12.826+0000 7fdab759e640 1 -- 172.21.15.88:0/1684579306 >> v1:172.21.15.88:6790/0 conn(0x7fdab002b570 legacy=0x7fdab00c7970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:12.816 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:12.829+0000 7fdab759e640 1 -- 172.21.15.88:0/1684579306 shutdown_connections 2024-03-29T10:41:12.816 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:12.829+0000 7fdab759e640 1 -- 172.21.15.88:0/1684579306 >> 172.21.15.88:0/1684579306 conn(0x7fdab00058f0 msgr2=0x7fdab0005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:41:12.816 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:12.829+0000 7fdab759e640 1 -- 172.21.15.88:0/1684579306 shutdown_connections 2024-03-29T10:41:12.816 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:12.829+0000 7fdab759e640 1 -- 172.21.15.88:0/1684579306 wait complete. 2024-03-29T10:41:12.816 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:12.830+0000 7fdab759e640 10 monclient: build_initial_monmap 2024-03-29T10:41:12.816 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:12.830+0000 7fdab759e640 10 monclient: monmap: 2024-03-29T10:41:12.816 INFO:teuthology.misc.health.smithi088.stderr:epoch 0 2024-03-29T10:41:12.816 INFO:teuthology.misc.health.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:41:12.816 INFO:teuthology.misc.health.smithi088.stderr:last_changed 0.000000 2024-03-29T10:41:12.816 INFO:teuthology.misc.health.smithi088.stderr:created 0.000000 2024-03-29T10:41:12.817 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 0 (unknown) 2024-03-29T10:41:12.817 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:41:12.817 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-03-29T10:41:12.817 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-29T10:41:12.817 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-c 2024-03-29T10:41:12.817 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:41:12.817 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:12.831+0000 7fdab759e640 1 librados: starting msgr at 2024-03-29T10:41:12.817 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:12.831+0000 7fdab759e640 1 librados: starting objecter 2024-03-29T10:41:12.817 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:12.831+0000 7fdab759e640 1 Processor -- start 2024-03-29T10:41:12.818 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:12.831+0000 7fdab759e640 1 -- start start 2024-03-29T10:41:12.818 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:12.831+0000 7fdab759e640 1 librados: setting wanted keys 2024-03-29T10:41:12.818 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:12.831+0000 7fdab759e640 1 librados: calling monclient init 2024-03-29T10:41:12.818 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:12.831+0000 7fdab759e640 10 monclient: init 2024-03-29T10:41:12.818 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:12.832+0000 7fdab759e640 10 monclient: _reopen_session rank -1 2024-03-29T10:41:12.818 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:12.832+0000 7fdab759e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:41:12.818 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:12.832+0000 7fdab759e640 10 monclient(hunting): picked mon.noname-b con 0x7fdab002b570 addr v1:172.21.15.196:6789/0 2024-03-29T10:41:12.818 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:12.832+0000 7fdab759e640 10 monclient(hunting): picked mon.noname-a con 0x7fdab00345e0 addr v1:172.21.15.88:6789/0 2024-03-29T10:41:12.819 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:12.832+0000 7fdab759e640 10 monclient(hunting): picked mon.noname-c con 0x7fdab00c24d0 addr v1:172.21.15.88:6790/0 2024-03-29T10:41:12.819 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:12.832+0000 7fdab759e640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdab00c3ff0 con 0x7fdab00345e0 2024-03-29T10:41:12.819 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:12.832+0000 7fdab759e640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdab00c53a0 con 0x7fdab002b570 2024-03-29T10:41:12.819 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:12.832+0000 7fdab759e640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdab002d920 con 0x7fdab00c24d0 2024-03-29T10:41:12.819 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:12.832+0000 7fdab759e640 10 monclient(hunting): _renew_subs 2024-03-29T10:41:12.819 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:12.832+0000 7fdab759e640 10 monclient(hunting): authenticate will time out at 795.068231s 2024-03-29T10:41:12.819 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:12.834+0000 7fdab659c640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7fdab002b570 0x7fdab00c4c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.88:38226/0 (socket says 172.21.15.88:38226) 2024-03-29T10:41:12.819 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:12.834+0000 7fdab659c640 1 -- 172.21.15.88:0/2113077132 learned_addr learned my addr 172.21.15.88:0/2113077132 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-29T10:41:12.821 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:12.837+0000 7fdaaeffd640 10 client.?.objecter ms_handle_connect 0x7fdab00345e0 2024-03-29T10:41:12.821 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:12.837+0000 7fdaaeffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:41:12.821 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:12.837+0000 7fdaaeffd640 10 client.?.objecter ms_handle_connect 0x7fdab002b570 2024-03-29T10:41:12.821 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:12.837+0000 7fdaaeffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:41:12.821 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:12.837+0000 7fdaaeffd640 1 -- 172.21.15.88:0/2113077132 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1654882610 0 0) 0x7fdab00c3ff0 con 0x7fdab00345e0 2024-03-29T10:41:12.822 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:12.837+0000 7fdaaeffd640 10 monclient(hunting): _negotiate 2024-03-29T10:41:12.822 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:12.837+0000 7fdaaeffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:41:12.822 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:12.837+0000 7fdaaeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:41:12.822 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:12.837+0000 7fdaaeffd640 10 monclient(hunting): my global_id is 4257 2024-03-29T10:41:12.822 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:12.838+0000 7fdaaeffd640 1 -- 172.21.15.88:0/2113077132 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fda98004eb0 con 0x7fdab00345e0 2024-03-29T10:41:12.822 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:12.838+0000 7fdaaeffd640 1 -- 172.21.15.88:0/2113077132 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2014205839 0 0) 0x7fdab00c53a0 con 0x7fdab002b570 2024-03-29T10:41:12.822 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:12.838+0000 7fdaaeffd640 10 monclient(hunting): _negotiate 2024-03-29T10:41:12.822 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:12.838+0000 7fdaaeffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:41:12.822 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:12.838+0000 7fdaaeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:41:12.822 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:12.838+0000 7fdaaeffd640 10 monclient(hunting): my global_id is 4249 2024-03-29T10:41:12.822 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:12.838+0000 7fdaaeffd640 1 -- 172.21.15.88:0/2113077132 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdab00c3ff0 con 0x7fdab002b570 2024-03-29T10:41:12.822 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:12.838+0000 7fdaaeffd640 10 client.?.objecter ms_handle_connect 0x7fdab00c24d0 2024-03-29T10:41:12.822 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:12.838+0000 7fdaaeffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:41:12.824 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:12.838+0000 7fdaaeffd640 1 -- 172.21.15.88:0/2113077132 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4180240900 0 0) 0x7fdab002d920 con 0x7fdab00c24d0 2024-03-29T10:41:12.824 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:12.838+0000 7fdaaeffd640 10 monclient(hunting): _negotiate 2024-03-29T10:41:12.824 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:12.838+0000 7fdaaeffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:41:12.824 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:12.838+0000 7fdaaeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:41:12.824 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:12.838+0000 7fdaaeffd640 10 monclient(hunting): my global_id is 4247 2024-03-29T10:41:12.824 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:12.838+0000 7fdaaeffd640 1 -- 172.21.15.88:0/2113077132 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdab00c53a0 con 0x7fdab00c24d0 2024-03-29T10:41:12.824 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:12.838+0000 7fdaaeffd640 1 -- 172.21.15.88:0/2113077132 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 733412480 0 0) 0x7fda98004eb0 con 0x7fdab00345e0 2024-03-29T10:41:12.824 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:12.838+0000 7fdaaeffd640 1 -- 172.21.15.88:0/2113077132 --> v1:172.21.15.88:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fda980066d0 con 0x7fdab00345e0 2024-03-29T10:41:12.824 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:12.838+0000 7fdaaeffd640 1 -- 172.21.15.88:0/2113077132 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3686821399 0 0) 0x7fdab00c3ff0 con 0x7fdab002b570 2024-03-29T10:41:12.824 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:12.838+0000 7fdaaeffd640 1 -- 172.21.15.88:0/2113077132 --> v1:172.21.15.196:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fda98004eb0 con 0x7fdab002b570 2024-03-29T10:41:12.824 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:12.839+0000 7fdaaeffd640 1 -- 172.21.15.88:0/2113077132 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7fdaa0001890 con 0x7fdab00345e0 2024-03-29T10:41:12.824 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:12.839+0000 7fdaaeffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:12.825 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:12.839+0000 7fdaaeffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:41:12.825 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:12.839+0000 7fdaaeffd640 10 monclient(hunting): dump: 2024-03-29T10:41:12.825 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:41:12.825 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:41:12.825 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:41:12.825 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:41:12.825 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:12.825 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:41:12.825 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:41:12.825 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:41:12.825 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:41:12.825 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:41:12.825 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:12.839+0000 7fdaaeffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:41:12.826 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:12.839+0000 7fdaaeffd640 1 -- 172.21.15.88:0/2113077132 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7fdaa8001820 con 0x7fdab002b570 2024-03-29T10:41:12.826 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:12.839+0000 7fdaaeffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:12.826 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:12.839+0000 7fdaaeffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T10:41:12.826 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:12.839+0000 7fdaaeffd640 10 monclient(hunting): dump: 2024-03-29T10:41:12.826 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:41:12.826 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:41:12.826 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:41:12.826 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:41:12.826 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:12.826 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:41:12.826 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:41:12.829 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:41:12.829 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:41:12.829 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:41:12.829 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:12.839+0000 7fdaaeffd640 1 -- 172.21.15.88:0/2113077132 <== mon.0 v1:172.21.15.88:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2659768126 0 0) 0x7fda980066d0 con 0x7fdab00345e0 2024-03-29T10:41:12.829 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:12.839+0000 7fdaaeffd640 1 -- 172.21.15.88:0/2113077132 >> v1:172.21.15.88:6790/0 conn(0x7fdab00c24d0 legacy=0x7fdab016bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:12.830 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:12.839+0000 7fdaaeffd640 1 -- 172.21.15.88:0/2113077132 >> v1:172.21.15.196:6789/0 conn(0x7fdab002b570 legacy=0x7fdab00c4c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:12.830 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:12.839+0000 7fdaaeffd640 10 monclient: _finish_hunting 0 2024-03-29T10:41:12.830 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:12.839+0000 7fdaaeffd640 1 monclient: found mon.a 2024-03-29T10:41:12.830 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:12.839+0000 7fdaaeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:41:12.830 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:12.839+0000 7fdaaeffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-03-29T10:41:12.830 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:12.839+0000 7fdaaeffd640 1 -- 172.21.15.88:0/2113077132 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdab002dc00 con 0x7fdab00345e0 2024-03-29T10:41:12.830 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:12.839+0000 7fdaaeffd640 10 monclient: _finish_auth 0 2024-03-29T10:41:12.830 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:12.839+0000 7fdaaeffd640 10 monclient: _check_auth_tickets 2024-03-29T10:41:12.830 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:12.839+0000 7fdaaeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:41:12.830 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:12.839+0000 7fdab759e640 5 monclient: authenticate success, global_id 4257 2024-03-29T10:41:12.830 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:12.839+0000 7fdab759e640 10 monclient: _renew_subs 2024-03-29T10:41:12.830 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:12.839+0000 7fdab759e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-03-29T10:41:12.831 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:12.839+0000 7fdab759e640 1 -- 172.21.15.88:0/2113077132 --> v1:172.21.15.88:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdab002dee0 con 0x7fdab00345e0 2024-03-29T10:41:12.831 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:12.839+0000 7fdab759e640 10 client.4257.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:41:12.831 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:12.839+0000 7fdab759e640 10 monclient: _renew_subs 2024-03-29T10:41:12.831 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:12.839+0000 7fdab759e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-03-29T10:41:12.831 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:12.839+0000 7fdab759e640 1 -- 172.21.15.88:0/2113077132 --> v1:172.21.15.88:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fdab00c28e0 con 0x7fdab00345e0 2024-03-29T10:41:12.831 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:12.839+0000 7fdab759e640 1 librados: init done 2024-03-29T10:41:12.831 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:12.839+0000 7fdaaeffd640 1 -- 172.21.15.88:0/2113077132 <== mon.0 v1:172.21.15.88:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdaa00047e0 con 0x7fdab00345e0 2024-03-29T10:41:12.831 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:12.839+0000 7fdaaeffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:41:12.831 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:12.839+0000 7fdaaeffd640 1 -- 172.21.15.88:0/2113077132 <== mon.0 v1:172.21.15.88:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7fdaa0004af0 con 0x7fdab00345e0 2024-03-29T10:41:12.831 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:12.839+0000 7fdaaeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:12.831 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:12.839+0000 7fdaaeffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:41:12.831 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:12.839+0000 7fdaaeffd640 10 monclient: dump: 2024-03-29T10:41:12.832 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:41:12.832 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:41:12.832 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:41:12.832 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:41:12.832 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:12.832 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:41:12.832 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:41:12.832 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:41:12.832 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:41:12.832 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:41:12.834 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:12.840+0000 7fdab759e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:41:12.834 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:12.840+0000 7fdab4d99640 4 set_mon_vals no callback set 2024-03-29T10:41:12.834 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:12.842+0000 7fdaaeffd640 1 -- 172.21.15.88:0/2113077132 <== mon.0 v1:172.21.15.88:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4235629220 0 0) 0x7fdaa001aac0 con 0x7fdab00345e0 2024-03-29T10:41:12.834 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:12.843+0000 7fdaaeffd640 1 -- 172.21.15.88:0/2113077132 <== mon.0 v1:172.21.15.88:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4257673582 0 0) 0x7fdaa0084980 con 0x7fdab00345e0 2024-03-29T10:41:12.834 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:12.843+0000 7fdaaeffd640 10 client.4257.objecter ms_dispatch 0x7fdab8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:41:12.834 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:12.843+0000 7fdaaeffd640 3 client.4257.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:41:12.834 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:12.843+0000 7fdaaeffd640 3 client.4257.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:41:12.834 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:12.843+0000 7fdaaeffd640 20 client.4257.objecter dump_active .. 0 homeless 2024-03-29T10:41:12.834 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:12.843+0000 7fdab759e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:41:12.834 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:12.843+0000 7fdab759e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-03-29T10:41:12.834 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:12.843+0000 7fdab759e640 1 -- 172.21.15.88:0/2113077132 --> v1:172.21.15.88:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdab0004cd0 con 0x7fdab00345e0 2024-03-29T10:41:12.835 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:12.843+0000 7fdab659c640 10 monclient: get_auth_request con 0x7fda9806dfd0 auth_method 0 2024-03-29T10:41:12.835 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:12.844+0000 7fdaaeffd640 10 client.4257.objecter ms_handle_connect 0x7fda9806dfd0 2024-03-29T10:41:12.837 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:12.853+0000 7fdaaeffd640 1 -- 172.21.15.88:0/2113077132 <== mon.0 v1:172.21.15.88:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fdaa0050260 con 0x7fdab00345e0 2024-03-29T10:41:12.837 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:12.854+0000 7fdaaeffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:41:12.837 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:12.854+0000 7fdaaeffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:41:13.159 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:13.175+0000 7fdab759e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:41:13.159 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:13.175+0000 7fdab759e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:41:13.159 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:13.175+0000 7fdab759e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-03-29T10:41:13.159 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:13.175+0000 7fdab759e640 1 -- 172.21.15.88:0/2113077132 --> v1:172.21.15.88:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdab002cd00 con 0x7fdab00345e0 2024-03-29T10:41:13.162 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:13.179+0000 7fdaaeffd640 1 -- 172.21.15.88:0/2113077132 <== mon.0 v1:172.21.15.88:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 344506374) 0x7fdaa0052190 con 0x7fdab00345e0 2024-03-29T10:41:13.162 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:13.179+0000 7fdaaeffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:41:13.162 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:13.179+0000 7fdaaeffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:41:13.163 INFO:teuthology.misc.health.smithi088.stdout: 2024-03-29T10:41:13.163 INFO:teuthology.misc.health.smithi088.stdout:{"epoch":10,"fsid":"869f5188-f92b-42dd-a46d-8c646e0388a9","created":"2024-03-29T10:40:40.285087+0000","modified":"2024-03-29T10:40:42.536230+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:40:42.536230+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a26c0c1c-8de0-438c-a3f6-3df10ea61574","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5de24915-8dec-49b6-8b16-f3170197c591","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ac054af3-d589-4565-9644-e6af75975c53","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3ec4f045-c714-4a22-bd1a-e07330b45ab3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"cddb9867-8e30-4951-9b13-3d1e78b9c45a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dc6d9911-785d-432f-8b53-56a1ccab302f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"77c88437-b587-496e-8b4d-1824ea8b5305","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ce963d96-5a8e-44c3-99de-c0dbebe234ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:41:13.166 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:13.183+0000 7fdab759e640 10 librados: watch_flush enter 2024-03-29T10:41:13.166 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:13.183+0000 7fdab759e640 10 librados: watch_flush exit 2024-03-29T10:41:13.167 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:13.183+0000 7fdab759e640 20 client.4257.objecter shutdown clearing up homeless session... 2024-03-29T10:41:13.167 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:13.183+0000 7fdab759e640 10 client.4257.objecter successfully canceled tick 2024-03-29T10:41:13.167 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:13.183+0000 7fdab759e640 1 -- 172.21.15.88:0/2113077132 >> v1:172.21.15.88:6816/1752861349 conn(0x7fda9806dfd0 legacy=0x7fda98070490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:13.167 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:13.183+0000 7fdab759e640 10 monclient: shutdown 2024-03-29T10:41:13.167 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:13.183+0000 7fdab759e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:41:13.167 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:13.183+0000 7fdab759e640 1 -- 172.21.15.88:0/2113077132 >> v1:172.21.15.88:6789/0 conn(0x7fdab00345e0 legacy=0x7fdab00c1dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:13.168 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:13.184+0000 7fdab759e640 1 -- 172.21.15.88:0/2113077132 shutdown_connections 2024-03-29T10:41:13.168 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:13.184+0000 7fdab759e640 1 -- 172.21.15.88:0/2113077132 >> 172.21.15.88:0/2113077132 conn(0x7fdab00058f0 msgr2=0x7fdab0033d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:41:13.168 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:13.184+0000 7fdab759e640 1 -- 172.21.15.88:0/2113077132 shutdown_connections 2024-03-29T10:41:13.168 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:13.184+0000 7fdab759e640 1 -- 172.21.15.88:0/2113077132 wait complete. 2024-03-29T10:41:13.168 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:13.184+0000 7fdab759e640 1 librados: shutdown 2024-03-29T10:41:13.190 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:41:19.997 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:41:20.152 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:20.168+0000 7fd39464e640 10 monclient: get_monmap_and_config 2024-03-29T10:41:20.152 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:20.168+0000 7fd39464e640 10 monclient: build_initial_monmap 2024-03-29T10:41:20.152 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:20.168+0000 7fd39464e640 10 monclient: monmap: 2024-03-29T10:41:20.152 INFO:teuthology.misc.health.smithi088.stderr:epoch 0 2024-03-29T10:41:20.153 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:41:20.153 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:41:20.169497+0000 2024-03-29T10:41:20.153 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:41:20.169497+0000 2024-03-29T10:41:20.153 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 0 (unknown) 2024-03-29T10:41:20.153 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:41:20.153 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-03-29T10:41:20.153 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-29T10:41:20.153 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-b 2024-03-29T10:41:20.153 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:41:20.154 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:20.169+0000 7fd39464e640 1 Processor -- start 2024-03-29T10:41:20.154 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:20.169+0000 7fd39464e640 1 -- start start 2024-03-29T10:41:20.154 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:20.169+0000 7fd39464e640 10 monclient: init 2024-03-29T10:41:20.154 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:20.170+0000 7fd39464e640 10 monclient: _reopen_session rank -1 2024-03-29T10:41:20.154 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:20.171+0000 7fd39464e640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:41:20.158 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:20.171+0000 7fd39464e640 10 monclient(hunting): picked mon.noname-b con 0x7fd38c071de0 addr v1:172.21.15.88:6790/0 2024-03-29T10:41:20.160 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:20.175+0000 7fd39464e640 10 monclient(hunting): picked mon.noname-a con 0x7fd38c0721c0 addr v1:172.21.15.88:6789/0 2024-03-29T10:41:20.163 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:20.177+0000 7fd3923ea640 1 --1- >> v1:172.21.15.88:6790/0 conn(0x7fd38c071de0 0x7fd38c0aed80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6790/0 says I am v1:172.21.15.88:58073/0 (socket says 172.21.15.88:58073) 2024-03-29T10:41:20.163 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:20.177+0000 7fd39464e640 10 monclient(hunting): picked mon.noname-c con 0x7fd38c145ae0 addr v1:172.21.15.196:6789/0 2024-03-29T10:41:20.163 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:20.177+0000 7fd39464e640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd38c145650 con 0x7fd38c0721c0 2024-03-29T10:41:20.163 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:20.177+0000 7fd3923ea640 1 -- 172.21.15.88:0/4062336757 learned_addr learned my addr 172.21.15.88:0/4062336757 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-29T10:41:20.163 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:20.177+0000 7fd39464e640 1 -- 172.21.15.88:0/4062336757 --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd38c0ae260 con 0x7fd38c145ae0 2024-03-29T10:41:20.163 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:20.177+0000 7fd39464e640 1 -- 172.21.15.88:0/4062336757 --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd38c1420d0 con 0x7fd38c071de0 2024-03-29T10:41:20.163 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:20.177+0000 7fd39464e640 10 monclient(hunting): _renew_subs 2024-03-29T10:41:20.163 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:20.177+0000 7fd39464e640 10 monclient(hunting): authenticate will time out at 802.413216s 2024-03-29T10:41:20.163 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:20.177+0000 7fd3913e8640 1 -- 172.21.15.88:0/4062336757 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3559218799 0 0) 0x7fd38c1420d0 con 0x7fd38c071de0 2024-03-29T10:41:20.163 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:20.177+0000 7fd3913e8640 10 monclient(hunting): _negotiate 2024-03-29T10:41:20.164 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:20.177+0000 7fd3913e8640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:41:20.164 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:20.177+0000 7fd3913e8640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:41:20.164 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:20.177+0000 7fd3913e8640 10 monclient(hunting): my global_id is 4250 2024-03-29T10:41:20.164 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:20.177+0000 7fd3913e8640 1 -- 172.21.15.88:0/4062336757 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd378003520 con 0x7fd38c071de0 2024-03-29T10:41:20.164 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:20.177+0000 7fd3913e8640 1 -- 172.21.15.88:0/4062336757 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2145135620 0 0) 0x7fd378003520 con 0x7fd38c071de0 2024-03-29T10:41:20.164 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:20.177+0000 7fd3913e8640 1 -- 172.21.15.88:0/4062336757 >> v1:172.21.15.196:6789/0 conn(0x7fd38c145ae0 legacy=0x7fd38c145f50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:41:20.167 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:20.180+0000 7fd3913e8640 1 -- 172.21.15.88:0/4062336757 >> v1:172.21.15.88:6789/0 conn(0x7fd38c0721c0 legacy=0x7fd38c13f400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:41:20.167 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:20.180+0000 7fd3913e8640 10 monclient: _finish_hunting 0 2024-03-29T10:41:20.167 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:20.180+0000 7fd3913e8640 1 monclient: found mon.noname-b 2024-03-29T10:41:20.167 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:20.180+0000 7fd3913e8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:41:20.167 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:20.180+0000 7fd3913e8640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.88:6790/0 2024-03-29T10:41:20.167 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:20.180+0000 7fd3913e8640 1 -- 172.21.15.88:0/4062336757 --> v1:172.21.15.88:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd38c0b0590 con 0x7fd38c071de0 2024-03-29T10:41:20.167 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:20.181+0000 7fd3913e8640 10 monclient: _finish_auth 0 2024-03-29T10:41:20.167 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:20.181+0000 7fd3913e8640 10 monclient: _check_auth_tickets 2024-03-29T10:41:20.167 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:20.181+0000 7fd3913e8640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:41:20.167 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:20.181+0000 7fd3913e8640 1 -- 172.21.15.88:0/4062336757 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7fd388001690 con 0x7fd38c071de0 2024-03-29T10:41:20.167 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:20.181+0000 7fd39464e640 5 monclient: authenticate success, global_id 4250 2024-03-29T10:41:20.168 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:20.181+0000 7fd39464e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:41:20.168 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:20.181+0000 7fd3913e8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:20.168 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:20.181+0000 7fd3913e8640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:41:20.168 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:20.181+0000 7fd3913e8640 10 monclient: dump: 2024-03-29T10:41:20.168 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:41:20.168 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:41:20.168 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:41:20.168 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:41:20.169 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:20.169 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:41:20.169 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:41:20.169 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:41:20.169 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:41:20.169 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:41:20.170 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:20.181+0000 7fd3913e8640 1 -- 172.21.15.88:0/4062336757 <== mon.2 v1:172.21.15.88:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd3880018b0 con 0x7fd38c071de0 2024-03-29T10:41:20.170 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:20.181+0000 7fd39464e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:41:20.170 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:20.181+0000 7fd3913e8640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:41:20.170 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:20.181+0000 7fd3913e8640 1 -- 172.21.15.88:0/4062336757 <== mon.2 v1:172.21.15.88:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7fd388001c40 con 0x7fd38c071de0 2024-03-29T10:41:20.170 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:20.181+0000 7fd39464e640 10 monclient: get_monmap_and_config success 2024-03-29T10:41:20.170 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:20.181+0000 7fd39464e640 4 set_mon_vals no callback set 2024-03-29T10:41:20.171 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:20.182+0000 7fd3913e8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:20.171 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:20.182+0000 7fd3913e8640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:41:20.171 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:20.182+0000 7fd3913e8640 10 monclient: dump: 2024-03-29T10:41:20.171 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:41:20.171 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:41:20.171 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:41:20.171 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:41:20.171 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:20.171 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:41:20.171 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:41:20.171 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:41:20.171 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:41:20.171 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:41:20.171 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:20.182+0000 7fd39464e640 10 monclient: shutdown 2024-03-29T10:41:20.172 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:20.182+0000 7fd39464e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:41:20.172 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:20.182+0000 7fd39464e640 1 -- 172.21.15.88:0/4062336757 >> v1:172.21.15.88:6790/0 conn(0x7fd38c071de0 legacy=0x7fd38c0aed80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:20.173 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:20.184+0000 7fd39464e640 1 -- 172.21.15.88:0/4062336757 shutdown_connections 2024-03-29T10:41:20.173 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:20.184+0000 7fd39464e640 1 -- 172.21.15.88:0/4062336757 >> 172.21.15.88:0/4062336757 conn(0x7fd38c06b660 msgr2=0x7fd38c06ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:41:20.174 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:20.185+0000 7fd39464e640 1 -- 172.21.15.88:0/4062336757 shutdown_connections 2024-03-29T10:41:20.174 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:20.186+0000 7fd39464e640 1 -- 172.21.15.88:0/4062336757 wait complete. 2024-03-29T10:41:20.174 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:20.187+0000 7fd39464e640 10 monclient: build_initial_monmap 2024-03-29T10:41:20.174 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:20.187+0000 7fd39464e640 10 monclient: monmap: 2024-03-29T10:41:20.174 INFO:teuthology.misc.health.smithi088.stderr:epoch 0 2024-03-29T10:41:20.174 INFO:teuthology.misc.health.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:41:20.174 INFO:teuthology.misc.health.smithi088.stderr:last_changed 0.000000 2024-03-29T10:41:20.174 INFO:teuthology.misc.health.smithi088.stderr:created 0.000000 2024-03-29T10:41:20.174 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 0 (unknown) 2024-03-29T10:41:20.174 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:41:20.174 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-03-29T10:41:20.174 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-29T10:41:20.174 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-c 2024-03-29T10:41:20.175 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:41:20.175 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:20.188+0000 7fd39464e640 1 librados: starting msgr at 2024-03-29T10:41:20.175 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:20.188+0000 7fd39464e640 1 librados: starting objecter 2024-03-29T10:41:20.175 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:20.188+0000 7fd39464e640 1 Processor -- start 2024-03-29T10:41:20.175 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:20.188+0000 7fd39464e640 1 -- start start 2024-03-29T10:41:20.175 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:20.188+0000 7fd39464e640 1 librados: setting wanted keys 2024-03-29T10:41:20.175 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:20.188+0000 7fd39464e640 1 librados: calling monclient init 2024-03-29T10:41:20.175 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:20.188+0000 7fd39464e640 10 monclient: init 2024-03-29T10:41:20.175 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:20.189+0000 7fd39464e640 10 monclient: _reopen_session rank -1 2024-03-29T10:41:20.175 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:20.189+0000 7fd39464e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:41:20.175 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:20.189+0000 7fd39464e640 10 monclient(hunting): picked mon.noname-b con 0x7fd38c071de0 addr v1:172.21.15.196:6789/0 2024-03-29T10:41:20.175 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:20.190+0000 7fd39464e640 10 monclient(hunting): picked mon.noname-a con 0x7fd38c145ae0 addr v1:172.21.15.88:6789/0 2024-03-29T10:41:20.175 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:20.190+0000 7fd39464e640 10 monclient(hunting): picked mon.noname-c con 0x7fd38c1324f0 addr v1:172.21.15.88:6790/0 2024-03-29T10:41:20.175 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:20.190+0000 7fd39464e640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd38c1301b0 con 0x7fd38c145ae0 2024-03-29T10:41:20.176 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:20.190+0000 7fd39464e640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd38c1d3cb0 con 0x7fd38c071de0 2024-03-29T10:41:20.176 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:20.190+0000 7fd39464e640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd38c136040 con 0x7fd38c1324f0 2024-03-29T10:41:20.176 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:20.190+0000 7fd39464e640 10 monclient(hunting): _renew_subs 2024-03-29T10:41:20.176 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:20.190+0000 7fd39464e640 10 monclient(hunting): authenticate will time out at 802.426109s 2024-03-29T10:41:20.176 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:20.193+0000 7fd392beb640 1 --1- >> v1:172.21.15.88:6790/0 conn(0x7fd38c1324f0 0x7fd38c12faa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6790/0 says I am v1:172.21.15.88:58195/0 (socket says 172.21.15.88:58195) 2024-03-29T10:41:20.176 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:20.193+0000 7fd392beb640 1 -- 172.21.15.88:0/3921418658 learned_addr learned my addr 172.21.15.88:0/3921418658 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-29T10:41:20.177 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:20.194+0000 7fd382ffd640 10 client.?.objecter ms_handle_connect 0x7fd38c1324f0 2024-03-29T10:41:20.178 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:20.194+0000 7fd382ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:41:20.178 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:20.194+0000 7fd382ffd640 1 -- 172.21.15.88:0/3921418658 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 32540842 0 0) 0x7fd38c136040 con 0x7fd38c1324f0 2024-03-29T10:41:20.178 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:20.194+0000 7fd382ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:41:20.178 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:20.194+0000 7fd382ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:41:20.178 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:20.194+0000 7fd382ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:41:20.178 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:20.194+0000 7fd382ffd640 10 monclient(hunting): my global_id is 4253 2024-03-29T10:41:20.178 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:20.194+0000 7fd382ffd640 1 -- 172.21.15.88:0/3921418658 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd370004eb0 con 0x7fd38c1324f0 2024-03-29T10:41:20.179 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:20.194+0000 7fd382ffd640 1 -- 172.21.15.88:0/3921418658 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 740611768 0 0) 0x7fd370004eb0 con 0x7fd38c1324f0 2024-03-29T10:41:20.179 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:20.194+0000 7fd382ffd640 1 -- 172.21.15.88:0/3921418658 --> v1:172.21.15.88:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd38c136040 con 0x7fd38c1324f0 2024-03-29T10:41:20.179 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:20.194+0000 7fd382ffd640 1 -- 172.21.15.88:0/3921418658 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7fd37c0017f0 con 0x7fd38c1324f0 2024-03-29T10:41:20.179 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:20.194+0000 7fd382ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:20.179 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:20.194+0000 7fd382ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:41:20.179 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:20.194+0000 7fd382ffd640 10 monclient(hunting): dump: 2024-03-29T10:41:20.179 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:41:20.179 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:41:20.179 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:41:20.179 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:41:20.179 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:20.179 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:41:20.179 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:41:20.179 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:41:20.180 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:41:20.180 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:41:20.180 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:20.195+0000 7fd382ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:41:20.180 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:20.195+0000 7fd382ffd640 1 -- 172.21.15.88:0/3921418658 <== mon.2 v1:172.21.15.88:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1282240856 0 0) 0x7fd38c136040 con 0x7fd38c1324f0 2024-03-29T10:41:20.180 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:20.195+0000 7fd382ffd640 1 -- 172.21.15.88:0/3921418658 >> v1:172.21.15.196:6789/0 conn(0x7fd38c071de0 legacy=0x7fd38c1d76b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:20.180 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:20.195+0000 7fd382ffd640 1 -- 172.21.15.88:0/3921418658 >> v1:172.21.15.88:6789/0 conn(0x7fd38c145ae0 legacy=0x7fd38c12f390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:41:20.181 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:20.197+0000 7fd382ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:41:20.181 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:20.197+0000 7fd382ffd640 1 monclient: found mon.c 2024-03-29T10:41:20.182 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:20.197+0000 7fd382ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:41:20.182 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:20.197+0000 7fd382ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-03-29T10:41:20.182 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:20.197+0000 7fd382ffd640 1 -- 172.21.15.88:0/3921418658 --> v1:172.21.15.88:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd38c138f70 con 0x7fd38c1324f0 2024-03-29T10:41:20.182 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:20.197+0000 7fd382ffd640 10 monclient: _finish_auth 0 2024-03-29T10:41:20.182 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:20.197+0000 7fd382ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:41:20.182 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:20.197+0000 7fd382ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:41:20.182 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:20.198+0000 7fd39464e640 5 monclient: authenticate success, global_id 4253 2024-03-29T10:41:20.182 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:20.198+0000 7fd39464e640 10 monclient: _renew_subs 2024-03-29T10:41:20.182 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:20.198+0000 7fd39464e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-03-29T10:41:20.182 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:20.198+0000 7fd39464e640 1 -- 172.21.15.88:0/3921418658 --> v1:172.21.15.88:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd38c139120 con 0x7fd38c1324f0 2024-03-29T10:41:20.182 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:20.198+0000 7fd382ffd640 1 -- 172.21.15.88:0/3921418658 <== mon.2 v1:172.21.15.88:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd37c001950 con 0x7fd38c1324f0 2024-03-29T10:41:20.182 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:20.198+0000 7fd382ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:41:20.182 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:20.198+0000 7fd383fff640 4 set_mon_vals no callback set 2024-03-29T10:41:20.183 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:20.198+0000 7fd39464e640 10 client.4253.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:41:20.183 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:20.198+0000 7fd382ffd640 1 -- 172.21.15.88:0/3921418658 <== mon.2 v1:172.21.15.88:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7fd37c004aa0 con 0x7fd38c1324f0 2024-03-29T10:41:20.183 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:20.198+0000 7fd39464e640 10 monclient: _renew_subs 2024-03-29T10:41:20.183 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:20.198+0000 7fd39464e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-03-29T10:41:20.183 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:20.198+0000 7fd39464e640 1 -- 172.21.15.88:0/3921418658 --> v1:172.21.15.88:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd38c1d7fa0 con 0x7fd38c1324f0 2024-03-29T10:41:20.184 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:20.199+0000 7fd39464e640 1 librados: init done 2024-03-29T10:41:20.184 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:20.199+0000 7fd382ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:20.186 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:20.200+0000 7fd382ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:41:20.186 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:20.200+0000 7fd382ffd640 10 monclient: dump: 2024-03-29T10:41:20.186 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:41:20.186 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:41:20.186 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:41:20.186 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:41:20.186 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:20.186 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:41:20.187 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:41:20.187 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:41:20.187 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:41:20.187 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:41:20.187 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:20.200+0000 7fd382ffd640 1 -- 172.21.15.88:0/3921418658 <== mon.2 v1:172.21.15.88:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4235629220 0 0) 0x7fd37c01aa70 con 0x7fd38c1324f0 2024-03-29T10:41:20.187 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:20.200+0000 7fd382ffd640 1 -- 172.21.15.88:0/3921418658 <== mon.2 v1:172.21.15.88:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4257673582 0 0) 0x7fd37c084b00 con 0x7fd38c1324f0 2024-03-29T10:41:20.187 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:20.201+0000 7fd382ffd640 10 client.4253.objecter ms_dispatch 0x7fd38c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:41:20.187 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:20.201+0000 7fd382ffd640 3 client.4253.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:41:20.187 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:20.201+0000 7fd382ffd640 3 client.4253.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:41:20.187 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:20.201+0000 7fd3923ea640 10 monclient: get_auth_request con 0x7fd37006dc00 auth_method 0 2024-03-29T10:41:20.187 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:20.201+0000 7fd382ffd640 20 client.4253.objecter dump_active .. 0 homeless 2024-03-29T10:41:20.188 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:20.201+0000 7fd382ffd640 10 client.4253.objecter ms_handle_connect 0x7fd37006dc00 2024-03-29T10:41:20.188 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:20.201+0000 7fd39464e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:41:20.188 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:20.201+0000 7fd39464e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:41:20.188 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:20.201+0000 7fd39464e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-03-29T10:41:20.189 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:20.201+0000 7fd39464e640 1 -- 172.21.15.88:0/3921418658 --> v1:172.21.15.88:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd38c1d73d0 con 0x7fd38c1324f0 2024-03-29T10:41:20.197 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:20.213+0000 7fd382ffd640 1 -- 172.21.15.88:0/3921418658 <== mon.2 v1:172.21.15.88:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd37c050210 con 0x7fd38c1324f0 2024-03-29T10:41:20.197 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:20.214+0000 7fd382ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:41:20.197 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:20.214+0000 7fd382ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:41:20.531 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:20.547+0000 7fd39464e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:41:20.531 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:20.547+0000 7fd39464e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:41:20.531 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:20.547+0000 7fd39464e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-03-29T10:41:20.531 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:20.547+0000 7fd39464e640 1 -- 172.21.15.88:0/3921418658 --> v1:172.21.15.88:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd38c075f10 con 0x7fd38c1324f0 2024-03-29T10:41:20.533 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:20.550+0000 7fd382ffd640 1 -- 172.21.15.88:0/3921418658 <== mon.2 v1:172.21.15.88:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 344506374) 0x7fd37c052310 con 0x7fd38c1324f0 2024-03-29T10:41:20.533 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:20.550+0000 7fd382ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:41:20.533 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:20.550+0000 7fd382ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:41:20.537 INFO:teuthology.misc.health.smithi088.stdout: 2024-03-29T10:41:20.537 INFO:teuthology.misc.health.smithi088.stdout:{"epoch":10,"fsid":"869f5188-f92b-42dd-a46d-8c646e0388a9","created":"2024-03-29T10:40:40.285087+0000","modified":"2024-03-29T10:40:42.536230+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:40:42.536230+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a26c0c1c-8de0-438c-a3f6-3df10ea61574","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5de24915-8dec-49b6-8b16-f3170197c591","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ac054af3-d589-4565-9644-e6af75975c53","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3ec4f045-c714-4a22-bd1a-e07330b45ab3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"cddb9867-8e30-4951-9b13-3d1e78b9c45a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dc6d9911-785d-432f-8b53-56a1ccab302f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"77c88437-b587-496e-8b4d-1824ea8b5305","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ce963d96-5a8e-44c3-99de-c0dbebe234ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:41:20.537 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:20.554+0000 7fd39464e640 10 librados: watch_flush enter 2024-03-29T10:41:20.539 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:20.555+0000 7fd39464e640 10 librados: watch_flush exit 2024-03-29T10:41:20.539 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:20.555+0000 7fd39464e640 20 client.4253.objecter shutdown clearing up homeless session... 2024-03-29T10:41:20.539 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:20.555+0000 7fd39464e640 10 client.4253.objecter successfully canceled tick 2024-03-29T10:41:20.539 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:20.556+0000 7fd39464e640 1 -- 172.21.15.88:0/3921418658 >> v1:172.21.15.88:6816/1752861349 conn(0x7fd37006dc00 legacy=0x7fd3700700c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:20.540 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:20.556+0000 7fd39464e640 10 monclient: shutdown 2024-03-29T10:41:20.540 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:20.556+0000 7fd39464e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:41:20.540 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:20.556+0000 7fd39464e640 1 -- 172.21.15.88:0/3921418658 >> v1:172.21.15.88:6790/0 conn(0x7fd38c1324f0 legacy=0x7fd38c12faa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:20.540 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:20.557+0000 7fd39464e640 1 -- 172.21.15.88:0/3921418658 shutdown_connections 2024-03-29T10:41:20.540 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:20.557+0000 7fd39464e640 1 -- 172.21.15.88:0/3921418658 >> 172.21.15.88:0/3921418658 conn(0x7fd38c06b660 msgr2=0x7fd38c0a9d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:41:20.541 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:20.558+0000 7fd39464e640 1 -- 172.21.15.88:0/3921418658 shutdown_connections 2024-03-29T10:41:20.541 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:20.558+0000 7fd39464e640 1 -- 172.21.15.88:0/3921418658 wait complete. 2024-03-29T10:41:20.544 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:20.561+0000 7fd39464e640 1 librados: shutdown 2024-03-29T10:41:20.563 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:41:27.370 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:41:27.518 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:27.534+0000 7fb5a9119640 10 monclient: get_monmap_and_config 2024-03-29T10:41:27.518 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:27.534+0000 7fb5a9119640 10 monclient: build_initial_monmap 2024-03-29T10:41:27.518 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:27.534+0000 7fb5a9119640 10 monclient: monmap: 2024-03-29T10:41:27.518 INFO:teuthology.misc.health.smithi088.stderr:epoch 0 2024-03-29T10:41:27.518 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:41:27.518 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:41:27.535440+0000 2024-03-29T10:41:27.518 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:41:27.535440+0000 2024-03-29T10:41:27.518 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 0 (unknown) 2024-03-29T10:41:27.519 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:41:27.519 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-03-29T10:41:27.519 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-29T10:41:27.519 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-b 2024-03-29T10:41:27.519 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:41:27.520 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:27.536+0000 7fb5a9119640 1 Processor -- start 2024-03-29T10:41:27.520 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:27.536+0000 7fb5a9119640 1 -- start start 2024-03-29T10:41:27.520 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:27.536+0000 7fb5a9119640 10 monclient: init 2024-03-29T10:41:27.521 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:27.537+0000 7fb5a9119640 10 monclient: _reopen_session rank -1 2024-03-29T10:41:27.521 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:27.537+0000 7fb5a9119640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:41:27.521 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:27.537+0000 7fb5a9119640 10 monclient(hunting): picked mon.noname-c con 0x7fb59c006a70 addr v1:172.21.15.196:6789/0 2024-03-29T10:41:27.521 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:27.538+0000 7fb5a9119640 10 monclient(hunting): picked mon.noname-b con 0x7fb59c02b100 addr v1:172.21.15.88:6790/0 2024-03-29T10:41:27.522 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:27.539+0000 7fb5a9119640 10 monclient(hunting): picked mon.noname-a con 0x7fb59c029c30 addr v1:172.21.15.88:6789/0 2024-03-29T10:41:27.522 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:27.539+0000 7fb5a9119640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb59c02a270 con 0x7fb59c029c30 2024-03-29T10:41:27.522 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:27.539+0000 7fb5a9119640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb59c02c440 con 0x7fb59c006a70 2024-03-29T10:41:27.522 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:27.539+0000 7fb5a9119640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb59c0c9440 con 0x7fb59c02b100 2024-03-29T10:41:27.522 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:27.539+0000 7fb5a9119640 10 monclient(hunting): _renew_subs 2024-03-29T10:41:27.522 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:27.539+0000 7fb5a9119640 10 monclient(hunting): authenticate will time out at 809.774938s 2024-03-29T10:41:27.524 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:27.541+0000 7fb5a37fe640 1 --1- >> v1:172.21.15.88:6790/0 conn(0x7fb59c02b100 0x7fb59c0cf100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6790/0 says I am v1:172.21.15.88:42011/0 (socket says 172.21.15.88:42011) 2024-03-29T10:41:27.525 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:27.541+0000 7fb5a37fe640 1 -- 172.21.15.88:0/3294834787 learned_addr learned my addr 172.21.15.88:0/3294834787 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-29T10:41:27.528 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:27.544+0000 7fb5a2ffd640 1 -- 172.21.15.88:0/3294834787 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1102881226 0 0) 0x7fb59c0c9440 con 0x7fb59c02b100 2024-03-29T10:41:27.528 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:27.544+0000 7fb5a2ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:41:27.528 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:27.544+0000 7fb5a2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:41:27.529 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:27.544+0000 7fb5a2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:41:27.529 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:27.544+0000 7fb5a2ffd640 10 monclient(hunting): my global_id is 4256 2024-03-29T10:41:27.529 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:27.544+0000 7fb5a2ffd640 1 -- 172.21.15.88:0/3294834787 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb594003520 con 0x7fb59c02b100 2024-03-29T10:41:27.529 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:27.545+0000 7fb5a2ffd640 1 -- 172.21.15.88:0/3294834787 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3545398821 0 0) 0x7fb594003520 con 0x7fb59c02b100 2024-03-29T10:41:27.529 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:27.546+0000 7fb5a2ffd640 1 -- 172.21.15.88:0/3294834787 >> v1:172.21.15.196:6789/0 conn(0x7fb59c006a70 legacy=0x7fb59c0c8d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:27.529 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:27.546+0000 7fb5a2ffd640 1 -- 172.21.15.88:0/3294834787 >> v1:172.21.15.88:6789/0 conn(0x7fb59c029c30 legacy=0x7fb59c0d74a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:27.530 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:27.546+0000 7fb5a2ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:41:27.530 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:27.546+0000 7fb5a2ffd640 1 monclient: found mon.noname-b 2024-03-29T10:41:27.530 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:27.546+0000 7fb5a2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:41:27.530 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:27.546+0000 7fb5a2ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.88:6790/0 2024-03-29T10:41:27.530 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:27.546+0000 7fb5a2ffd640 1 -- 172.21.15.88:0/3294834787 --> v1:172.21.15.88:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb59c0c95d0 con 0x7fb59c02b100 2024-03-29T10:41:27.530 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:27.546+0000 7fb5a2ffd640 10 monclient: _finish_auth 0 2024-03-29T10:41:27.530 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:27.546+0000 7fb5a2ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:41:27.530 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:27.546+0000 7fb5a2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:41:27.530 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:27.546+0000 7fb5a2ffd640 1 -- 172.21.15.88:0/3294834787 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7fb590001690 con 0x7fb59c02b100 2024-03-29T10:41:27.530 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:27.546+0000 7fb5a2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:27.530 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:27.546+0000 7fb5a2ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:41:27.530 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:27.546+0000 7fb5a2ffd640 10 monclient: dump: 2024-03-29T10:41:27.530 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:41:27.530 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:41:27.531 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:41:27.531 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:41:27.531 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:27.531 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:41:27.531 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:41:27.531 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:41:27.531 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:41:27.531 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:41:27.531 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:27.546+0000 7fb5a9119640 5 monclient: authenticate success, global_id 4256 2024-03-29T10:41:27.531 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:27.546+0000 7fb5a9119640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:41:27.531 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:27.546+0000 7fb5a2ffd640 1 -- 172.21.15.88:0/3294834787 <== mon.2 v1:172.21.15.88:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb590001860 con 0x7fb59c02b100 2024-03-29T10:41:27.531 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:27.546+0000 7fb5a2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:41:27.531 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:27.546+0000 7fb5a2ffd640 1 -- 172.21.15.88:0/3294834787 <== mon.2 v1:172.21.15.88:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7fb590001cb0 con 0x7fb59c02b100 2024-03-29T10:41:27.531 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:27.546+0000 7fb5a2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:27.532 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:27.546+0000 7fb5a2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:41:27.532 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:27.546+0000 7fb5a2ffd640 10 monclient: dump: 2024-03-29T10:41:27.532 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:41:27.532 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:41:27.532 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:41:27.532 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:41:27.532 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:27.532 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:41:27.532 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:41:27.532 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:41:27.532 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:41:27.532 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:41:27.533 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:27.546+0000 7fb5a9119640 10 monclient: get_monmap_and_config success 2024-03-29T10:41:27.533 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:27.546+0000 7fb5a9119640 4 set_mon_vals no callback set 2024-03-29T10:41:27.533 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:27.547+0000 7fb5a9119640 10 monclient: shutdown 2024-03-29T10:41:27.533 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:27.547+0000 7fb5a9119640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:41:27.533 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:27.547+0000 7fb5a9119640 1 -- 172.21.15.88:0/3294834787 >> v1:172.21.15.88:6790/0 conn(0x7fb59c02b100 legacy=0x7fb59c0cf100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:27.535 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:27.550+0000 7fb5a9119640 1 -- 172.21.15.88:0/3294834787 shutdown_connections 2024-03-29T10:41:27.535 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:27.550+0000 7fb5a9119640 1 -- 172.21.15.88:0/3294834787 >> 172.21.15.88:0/3294834787 conn(0x7fb59c0058f0 msgr2=0x7fb59c005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:41:27.535 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:27.552+0000 7fb5a9119640 1 -- 172.21.15.88:0/3294834787 shutdown_connections 2024-03-29T10:41:27.539 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:27.553+0000 7fb5a9119640 1 -- 172.21.15.88:0/3294834787 wait complete. 2024-03-29T10:41:27.539 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:27.554+0000 7fb5a9119640 10 monclient: build_initial_monmap 2024-03-29T10:41:27.540 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:27.554+0000 7fb5a9119640 10 monclient: monmap: 2024-03-29T10:41:27.540 INFO:teuthology.misc.health.smithi088.stderr:epoch 0 2024-03-29T10:41:27.540 INFO:teuthology.misc.health.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:41:27.540 INFO:teuthology.misc.health.smithi088.stderr:last_changed 0.000000 2024-03-29T10:41:27.540 INFO:teuthology.misc.health.smithi088.stderr:created 0.000000 2024-03-29T10:41:27.540 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 0 (unknown) 2024-03-29T10:41:27.540 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:41:27.540 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-03-29T10:41:27.540 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-29T10:41:27.540 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-c 2024-03-29T10:41:27.540 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:41:27.540 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:27.554+0000 7fb5a9119640 1 librados: starting msgr at 2024-03-29T10:41:27.540 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:27.554+0000 7fb5a9119640 1 librados: starting objecter 2024-03-29T10:41:27.540 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:27.554+0000 7fb5a9119640 1 Processor -- start 2024-03-29T10:41:27.541 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:27.554+0000 7fb5a9119640 1 -- start start 2024-03-29T10:41:27.541 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:27.554+0000 7fb5a9119640 1 librados: setting wanted keys 2024-03-29T10:41:27.541 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:27.554+0000 7fb5a9119640 1 librados: calling monclient init 2024-03-29T10:41:27.541 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:27.554+0000 7fb5a9119640 10 monclient: init 2024-03-29T10:41:27.541 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:27.555+0000 7fb5a9119640 10 monclient: _reopen_session rank -1 2024-03-29T10:41:27.541 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:27.555+0000 7fb5a9119640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:41:27.541 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:27.555+0000 7fb5a9119640 10 monclient(hunting): picked mon.noname-a con 0x7fb59c006a70 addr v1:172.21.15.88:6789/0 2024-03-29T10:41:27.541 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:27.555+0000 7fb5a9119640 10 monclient(hunting): picked mon.noname-b con 0x7fb59c029c30 addr v1:172.21.15.196:6789/0 2024-03-29T10:41:27.541 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:27.558+0000 7fb5a9119640 10 monclient(hunting): picked mon.noname-c con 0x7fb59c0ccbe0 addr v1:172.21.15.88:6790/0 2024-03-29T10:41:27.541 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:27.558+0000 7fb5a9119640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb59c16cd20 con 0x7fb59c006a70 2024-03-29T10:41:27.542 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:27.558+0000 7fb5a9119640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb59c02edc0 con 0x7fb59c029c30 2024-03-29T10:41:27.542 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:27.558+0000 7fb5a9119640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb59c16d8c0 con 0x7fb59c0ccbe0 2024-03-29T10:41:27.542 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:27.558+0000 7fb5a9119640 10 monclient(hunting): _renew_subs 2024-03-29T10:41:27.542 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:27.558+0000 7fb5a9119640 10 monclient(hunting): authenticate will time out at 809.794189s 2024-03-29T10:41:27.543 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:27.559+0000 7fb5a3fff640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fb59c006a70 0x7fb59c16d1b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:33291/0 (socket says 172.21.15.88:33291) 2024-03-29T10:41:27.543 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:27.559+0000 7fb5a3fff640 1 -- 172.21.15.88:0/594652706 learned_addr learned my addr 172.21.15.88:0/594652706 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-29T10:41:27.543 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:27.560+0000 7fb5a0ff9640 10 client.?.objecter ms_handle_connect 0x7fb59c006a70 2024-03-29T10:41:27.543 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:27.560+0000 7fb5a0ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:41:27.544 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:27.560+0000 7fb5a0ff9640 1 -- 172.21.15.88:0/594652706 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3755950494 0 0) 0x7fb59c16cd20 con 0x7fb59c006a70 2024-03-29T10:41:27.544 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:27.560+0000 7fb5a0ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:41:27.544 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:27.560+0000 7fb5a0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:41:27.544 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:27.560+0000 7fb5a0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:41:27.544 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:27.560+0000 7fb5a0ff9640 10 monclient(hunting): my global_id is 4263 2024-03-29T10:41:27.544 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:27.560+0000 7fb5a0ff9640 1 -- 172.21.15.88:0/594652706 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb588004eb0 con 0x7fb59c006a70 2024-03-29T10:41:27.545 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:27.561+0000 7fb5a0ff9640 1 -- 172.21.15.88:0/594652706 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2125023644 0 0) 0x7fb588004eb0 con 0x7fb59c006a70 2024-03-29T10:41:27.545 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:27.561+0000 7fb5a0ff9640 1 -- 172.21.15.88:0/594652706 --> v1:172.21.15.88:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb59c16cd20 con 0x7fb59c006a70 2024-03-29T10:41:27.545 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:27.561+0000 7fb5a0ff9640 1 -- 172.21.15.88:0/594652706 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7fb5a404fdd0 con 0x7fb59c006a70 2024-03-29T10:41:27.545 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:27.561+0000 7fb5a0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:27.545 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:27.561+0000 7fb5a0ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:41:27.545 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:27.561+0000 7fb5a0ff9640 10 monclient(hunting): dump: 2024-03-29T10:41:27.545 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:41:27.545 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:41:27.545 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:41:27.546 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:41:27.546 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:27.546 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:41:27.546 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:41:27.546 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:41:27.546 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:41:27.546 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:41:27.546 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:27.561+0000 7fb5a0ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:41:27.546 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:27.561+0000 7fb5a0ff9640 10 client.?.objecter ms_handle_connect 0x7fb59c0ccbe0 2024-03-29T10:41:27.546 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:27.561+0000 7fb5a0ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:41:27.546 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:27.561+0000 7fb5a0ff9640 1 -- 172.21.15.88:0/594652706 <== mon.0 v1:172.21.15.88:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2796059224 0 0) 0x7fb59c16cd20 con 0x7fb59c006a70 2024-03-29T10:41:27.546 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:27.561+0000 7fb5a0ff9640 1 -- 172.21.15.88:0/594652706 >> v1:172.21.15.88:6790/0 conn(0x7fb59c0ccbe0 legacy=0x7fb59c174070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:27.546 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:27.561+0000 7fb5a0ff9640 1 -- 172.21.15.88:0/594652706 >> v1:172.21.15.196:6789/0 conn(0x7fb59c029c30 legacy=0x7fb59c02d240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:41:27.547 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:27.562+0000 7fb5a0ff9640 10 monclient: _finish_hunting 0 2024-03-29T10:41:27.547 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:27.562+0000 7fb5a0ff9640 1 monclient: found mon.a 2024-03-29T10:41:27.547 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:27.562+0000 7fb5a0ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:41:27.547 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:27.562+0000 7fb5a0ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-03-29T10:41:27.547 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:27.562+0000 7fb5a0ff9640 1 -- 172.21.15.88:0/594652706 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb59c02d950 con 0x7fb59c006a70 2024-03-29T10:41:27.548 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:27.562+0000 7fb5a0ff9640 10 monclient: _finish_auth 0 2024-03-29T10:41:27.548 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:27.562+0000 7fb5a0ff9640 10 monclient: _check_auth_tickets 2024-03-29T10:41:27.548 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:27.562+0000 7fb5a0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:41:27.548 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:27.562+0000 7fb5a9119640 5 monclient: authenticate success, global_id 4263 2024-03-29T10:41:27.548 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:27.562+0000 7fb5a9119640 10 monclient: _renew_subs 2024-03-29T10:41:27.548 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:27.562+0000 7fb5a9119640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-03-29T10:41:27.548 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:27.562+0000 7fb5a9119640 1 -- 172.21.15.88:0/594652706 --> v1:172.21.15.88:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb59c02dca0 con 0x7fb59c006a70 2024-03-29T10:41:27.548 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:27.562+0000 7fb5a9119640 10 client.4263.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:41:27.548 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:27.562+0000 7fb5a9119640 10 monclient: _renew_subs 2024-03-29T10:41:27.548 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:27.562+0000 7fb5a9119640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-03-29T10:41:27.548 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:27.562+0000 7fb5a9119640 1 -- 172.21.15.88:0/594652706 --> v1:172.21.15.88:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb59c02df50 con 0x7fb59c006a70 2024-03-29T10:41:27.548 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:27.562+0000 7fb5a9119640 1 librados: init done 2024-03-29T10:41:27.548 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:27.563+0000 7fb5a0ff9640 1 -- 172.21.15.88:0/594652706 <== mon.0 v1:172.21.15.88:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb5a404f400 con 0x7fb59c006a70 2024-03-29T10:41:27.548 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:27.563+0000 7fb5a0ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:41:27.548 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:27.563+0000 7fb5a0ff9640 1 -- 172.21.15.88:0/594652706 <== mon.0 v1:172.21.15.88:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7fb5a4075f10 con 0x7fb59c006a70 2024-03-29T10:41:27.549 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:27.563+0000 7fb5a0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:27.549 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:27.563+0000 7fb5a0ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:41:27.549 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:27.563+0000 7fb5a0ff9640 10 monclient: dump: 2024-03-29T10:41:27.549 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:41:27.549 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:41:27.549 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:41:27.549 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:41:27.549 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:27.549 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:41:27.549 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:41:27.549 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:41:27.549 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:41:27.549 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:41:27.549 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:27.563+0000 7fb5a27fc640 4 set_mon_vals no callback set 2024-03-29T10:41:27.551 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:27.565+0000 7fb5a0ff9640 1 -- 172.21.15.88:0/594652706 <== mon.0 v1:172.21.15.88:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4235629220 0 0) 0x7fb5a4067bf0 con 0x7fb59c006a70 2024-03-29T10:41:27.551 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:27.565+0000 7fb5a0ff9640 1 -- 172.21.15.88:0/594652706 <== mon.0 v1:172.21.15.88:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4257673582 0 0) 0x7fb5a40ff620 con 0x7fb59c006a70 2024-03-29T10:41:27.551 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:27.565+0000 7fb5a0ff9640 10 client.4263.objecter ms_dispatch 0x7fb5a4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:41:27.551 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:27.565+0000 7fb5a0ff9640 3 client.4263.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:41:27.551 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:27.565+0000 7fb5a0ff9640 3 client.4263.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:41:27.551 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:27.565+0000 7fb5a0ff9640 20 client.4263.objecter dump_active .. 0 homeless 2024-03-29T10:41:27.551 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:27.566+0000 7fb5a9119640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:41:27.551 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:27.566+0000 7fb5a9119640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:41:27.551 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:27.566+0000 7fb5a9119640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-03-29T10:41:27.551 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:27.566+0000 7fb5a9119640 1 -- 172.21.15.88:0/594652706 --> v1:172.21.15.88:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb584005080 con 0x7fb59c006a70 2024-03-29T10:41:27.551 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:27.566+0000 7fb5a37fe640 10 monclient: get_auth_request con 0x7fb58806db10 auth_method 0 2024-03-29T10:41:27.551 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:27.566+0000 7fb5a0ff9640 10 client.4263.objecter ms_handle_connect 0x7fb58806db10 2024-03-29T10:41:27.559 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:27.576+0000 7fb5a0ff9640 1 -- 172.21.15.88:0/594652706 <== mon.0 v1:172.21.15.88:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb5a40caf00 con 0x7fb59c006a70 2024-03-29T10:41:27.559 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:27.576+0000 7fb5a0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:41:27.559 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:27.576+0000 7fb5a0ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:41:27.882 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:27.898+0000 7fb5a9119640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:41:27.882 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:27.898+0000 7fb5a9119640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:41:27.882 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:27.898+0000 7fb5a9119640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-03-29T10:41:27.882 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:27.898+0000 7fb5a9119640 1 -- 172.21.15.88:0/594652706 --> v1:172.21.15.88:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb584005330 con 0x7fb59c006a70 2024-03-29T10:41:27.883 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:27.899+0000 7fb5a0ff9640 1 -- 172.21.15.88:0/594652706 <== mon.0 v1:172.21.15.88:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 344506374) 0x7fb5a4067f20 con 0x7fb59c006a70 2024-03-29T10:41:27.883 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:27.899+0000 7fb5a0ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:41:27.883 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:27.899+0000 7fb5a0ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:41:27.883 INFO:teuthology.misc.health.smithi088.stdout: 2024-03-29T10:41:27.883 INFO:teuthology.misc.health.smithi088.stdout:{"epoch":10,"fsid":"869f5188-f92b-42dd-a46d-8c646e0388a9","created":"2024-03-29T10:40:40.285087+0000","modified":"2024-03-29T10:40:42.536230+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:40:42.536230+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a26c0c1c-8de0-438c-a3f6-3df10ea61574","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5de24915-8dec-49b6-8b16-f3170197c591","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ac054af3-d589-4565-9644-e6af75975c53","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3ec4f045-c714-4a22-bd1a-e07330b45ab3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"cddb9867-8e30-4951-9b13-3d1e78b9c45a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dc6d9911-785d-432f-8b53-56a1ccab302f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"77c88437-b587-496e-8b4d-1824ea8b5305","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ce963d96-5a8e-44c3-99de-c0dbebe234ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:41:27.888 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:27.904+0000 7fb5a9119640 10 librados: watch_flush enter 2024-03-29T10:41:27.889 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:27.905+0000 7fb5a9119640 10 librados: watch_flush exit 2024-03-29T10:41:27.889 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:27.905+0000 7fb5a9119640 20 client.4263.objecter shutdown clearing up homeless session... 2024-03-29T10:41:27.889 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:27.905+0000 7fb5a9119640 10 client.4263.objecter successfully canceled tick 2024-03-29T10:41:27.891 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:27.906+0000 7fb5a9119640 1 -- 172.21.15.88:0/594652706 >> v1:172.21.15.88:6816/1752861349 conn(0x7fb58806db10 legacy=0x7fb58806ffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:27.891 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:27.907+0000 7fb5a9119640 10 monclient: shutdown 2024-03-29T10:41:27.891 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:27.907+0000 7fb5a9119640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:41:27.891 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:27.907+0000 7fb5a9119640 1 -- 172.21.15.88:0/594652706 >> v1:172.21.15.88:6789/0 conn(0x7fb59c006a70 legacy=0x7fb59c16d1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:27.892 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:27.907+0000 7fb5a9119640 1 -- 172.21.15.88:0/594652706 shutdown_connections 2024-03-29T10:41:27.892 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:27.907+0000 7fb5a9119640 1 -- 172.21.15.88:0/594652706 >> 172.21.15.88:0/594652706 conn(0x7fb59c0058f0 msgr2=0x7fb59c0c3f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:41:27.892 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:27.907+0000 7fb5a9119640 1 -- 172.21.15.88:0/594652706 shutdown_connections 2024-03-29T10:41:27.892 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:27.907+0000 7fb5a9119640 1 -- 172.21.15.88:0/594652706 wait complete. 2024-03-29T10:41:27.892 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:27.907+0000 7fb5a9119640 1 librados: shutdown 2024-03-29T10:41:27.913 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:41:34.721 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:41:34.883 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:34.899+0000 7f606e12b640 10 monclient: get_monmap_and_config 2024-03-29T10:41:34.883 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:34.899+0000 7f606e12b640 10 monclient: build_initial_monmap 2024-03-29T10:41:34.883 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:34.899+0000 7f606e12b640 10 monclient: monmap: 2024-03-29T10:41:34.883 INFO:teuthology.misc.health.smithi088.stderr:epoch 0 2024-03-29T10:41:34.883 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:41:34.883 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:41:34.900411+0000 2024-03-29T10:41:34.883 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:41:34.900411+0000 2024-03-29T10:41:34.883 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 0 (unknown) 2024-03-29T10:41:34.884 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:41:34.884 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-03-29T10:41:34.884 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-29T10:41:34.884 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-b 2024-03-29T10:41:34.884 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:41:34.884 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:34.900+0000 7f606e12b640 1 Processor -- start 2024-03-29T10:41:34.886 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:34.900+0000 7f606e12b640 1 -- start start 2024-03-29T10:41:34.886 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:34.900+0000 7f606e12b640 10 monclient: init 2024-03-29T10:41:34.886 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:34.901+0000 7f606e12b640 10 monclient: _reopen_session rank -1 2024-03-29T10:41:34.886 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:34.901+0000 7f606e12b640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:41:34.886 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:34.901+0000 7f606e12b640 10 monclient(hunting): picked mon.noname-b con 0x7f60680b0770 addr v1:172.21.15.88:6790/0 2024-03-29T10:41:34.886 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:34.901+0000 7f606e12b640 10 monclient(hunting): picked mon.noname-c con 0x7f606814a700 addr v1:172.21.15.196:6789/0 2024-03-29T10:41:34.886 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:34.901+0000 7f606e12b640 10 monclient(hunting): picked mon.noname-a con 0x7f60680ac5a0 addr v1:172.21.15.88:6789/0 2024-03-29T10:41:34.886 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:34.901+0000 7f606e12b640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60680aba70 con 0x7f60680ac5a0 2024-03-29T10:41:34.886 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:34.901+0000 7f606e12b640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60680aecb0 con 0x7f606814a700 2024-03-29T10:41:34.886 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:34.901+0000 7f606e12b640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60680af150 con 0x7f60680b0770 2024-03-29T10:41:34.887 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:34.901+0000 7f606e12b640 10 monclient(hunting): _renew_subs 2024-03-29T10:41:34.887 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:34.901+0000 7f606e12b640 10 monclient(hunting): authenticate will time out at 817.137343s 2024-03-29T10:41:34.890 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:34.905+0000 7f6067fff640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f60680ac5a0 0x7f6068151f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:55805/0 (socket says 172.21.15.88:55805) 2024-03-29T10:41:34.890 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:34.905+0000 7f6067fff640 1 -- 172.21.15.88:0/669891434 learned_addr learned my addr 172.21.15.88:0/669891434 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-29T10:41:34.890 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:34.906+0000 7f60667fc640 1 -- 172.21.15.88:0/669891434 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 741032948 0 0) 0x7f60680aba70 con 0x7f60680ac5a0 2024-03-29T10:41:34.890 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:34.906+0000 7f60667fc640 10 monclient(hunting): _negotiate 2024-03-29T10:41:34.890 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:34.906+0000 7f60667fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:41:34.890 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:34.906+0000 7f60667fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:41:34.890 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:34.906+0000 7f60667fc640 10 monclient(hunting): my global_id is 4266 2024-03-29T10:41:34.890 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:34.906+0000 7f60667fc640 1 -- 172.21.15.88:0/669891434 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6054003520 con 0x7f60680ac5a0 2024-03-29T10:41:34.890 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:34.906+0000 7f60667fc640 1 -- 172.21.15.88:0/669891434 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2943292488 0 0) 0x7f6054003520 con 0x7f60680ac5a0 2024-03-29T10:41:34.890 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:34.906+0000 7f60667fc640 1 -- 172.21.15.88:0/669891434 >> v1:172.21.15.88:6790/0 conn(0x7f60680b0770 legacy=0x7f6068149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:41:34.890 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:34.906+0000 7f60667fc640 1 -- 172.21.15.88:0/669891434 >> v1:172.21.15.196:6789/0 conn(0x7f606814a700 legacy=0x7f60680abe90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:41:34.891 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:34.907+0000 7f60667fc640 10 monclient: _finish_hunting 0 2024-03-29T10:41:34.891 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:34.907+0000 7f60667fc640 1 monclient: found mon.noname-a 2024-03-29T10:41:34.891 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:34.907+0000 7f60667fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:41:34.891 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:34.907+0000 7f60667fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.88:6789/0 2024-03-29T10:41:34.891 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:34.907+0000 7f60667fc640 1 -- 172.21.15.88:0/669891434 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60680ac9d0 con 0x7f60680ac5a0 2024-03-29T10:41:34.891 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:34.907+0000 7f60667fc640 10 monclient: _finish_auth 0 2024-03-29T10:41:34.891 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:34.907+0000 7f60667fc640 10 monclient: _check_auth_tickets 2024-03-29T10:41:34.891 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:34.907+0000 7f60667fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:41:34.891 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:34.907+0000 7f60667fc640 1 -- 172.21.15.88:0/669891434 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f6050001690 con 0x7f60680ac5a0 2024-03-29T10:41:34.891 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:34.907+0000 7f60667fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:34.892 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:34.907+0000 7f60667fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:41:34.892 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:34.907+0000 7f60667fc640 10 monclient: dump: 2024-03-29T10:41:34.892 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:41:34.892 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:41:34.892 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:41:34.892 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:41:34.892 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:34.892 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:41:34.892 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:41:34.892 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:41:34.892 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:41:34.892 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:41:34.892 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:34.907+0000 7f60667fc640 1 -- 172.21.15.88:0/669891434 <== mon.0 v1:172.21.15.88:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6050001920 con 0x7f60680ac5a0 2024-03-29T10:41:34.893 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:34.907+0000 7f60667fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:41:34.893 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:34.907+0000 7f60667fc640 1 -- 172.21.15.88:0/669891434 <== mon.0 v1:172.21.15.88:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f6050001cf0 con 0x7f60680ac5a0 2024-03-29T10:41:34.893 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:34.907+0000 7f60667fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:34.893 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:34.907+0000 7f60667fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:41:34.893 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:34.907+0000 7f60667fc640 10 monclient: dump: 2024-03-29T10:41:34.893 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:41:34.893 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:41:34.893 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:41:34.893 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:41:34.893 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:34.893 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:41:34.893 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:41:34.893 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:41:34.893 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:41:34.893 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:41:34.895 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:34.907+0000 7f606e12b640 5 monclient: authenticate success, global_id 4266 2024-03-29T10:41:34.895 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:34.907+0000 7f606e12b640 10 monclient: get_monmap_and_config success 2024-03-29T10:41:34.895 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:34.907+0000 7f606e12b640 4 set_mon_vals no callback set 2024-03-29T10:41:34.895 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:34.908+0000 7f606e12b640 10 monclient: shutdown 2024-03-29T10:41:34.895 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:34.908+0000 7f606e12b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:41:34.895 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:34.908+0000 7f606e12b640 1 -- 172.21.15.88:0/669891434 >> v1:172.21.15.88:6789/0 conn(0x7f60680ac5a0 legacy=0x7f6068151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:34.895 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:34.908+0000 7f606e12b640 1 -- 172.21.15.88:0/669891434 shutdown_connections 2024-03-29T10:41:34.895 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:34.908+0000 7f606e12b640 1 -- 172.21.15.88:0/669891434 >> 172.21.15.88:0/669891434 conn(0x7f606806b660 msgr2=0x7f606806ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:41:34.896 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:34.909+0000 7f606e12b640 1 -- 172.21.15.88:0/669891434 shutdown_connections 2024-03-29T10:41:34.896 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:34.909+0000 7f606e12b640 1 -- 172.21.15.88:0/669891434 wait complete. 2024-03-29T10:41:34.896 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:34.909+0000 7f606e12b640 10 monclient: build_initial_monmap 2024-03-29T10:41:34.896 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:34.909+0000 7f606e12b640 10 monclient: monmap: 2024-03-29T10:41:34.896 INFO:teuthology.misc.health.smithi088.stderr:epoch 0 2024-03-29T10:41:34.896 INFO:teuthology.misc.health.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:41:34.896 INFO:teuthology.misc.health.smithi088.stderr:last_changed 0.000000 2024-03-29T10:41:34.896 INFO:teuthology.misc.health.smithi088.stderr:created 0.000000 2024-03-29T10:41:34.896 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 0 (unknown) 2024-03-29T10:41:34.896 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:41:34.896 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-03-29T10:41:34.896 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-29T10:41:34.896 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-c 2024-03-29T10:41:34.896 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:41:34.896 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:34.910+0000 7f606e12b640 1 librados: starting msgr at 2024-03-29T10:41:34.896 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:34.910+0000 7f606e12b640 1 librados: starting objecter 2024-03-29T10:41:34.897 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:34.910+0000 7f606e12b640 1 Processor -- start 2024-03-29T10:41:34.897 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:34.910+0000 7f606e12b640 1 -- start start 2024-03-29T10:41:34.897 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:34.910+0000 7f606e12b640 1 librados: setting wanted keys 2024-03-29T10:41:34.897 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:34.910+0000 7f606e12b640 1 librados: calling monclient init 2024-03-29T10:41:34.897 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:34.910+0000 7f606e12b640 10 monclient: init 2024-03-29T10:41:34.897 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:34.911+0000 7f606e12b640 10 monclient: _reopen_session rank -1 2024-03-29T10:41:34.897 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:34.911+0000 7f606e12b640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:41:34.897 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:34.911+0000 7f606e12b640 10 monclient(hunting): picked mon.noname-b con 0x7f60680ac5a0 addr v1:172.21.15.196:6789/0 2024-03-29T10:41:34.897 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:34.911+0000 7f606e12b640 10 monclient(hunting): picked mon.noname-c con 0x7f60680b0770 addr v1:172.21.15.88:6790/0 2024-03-29T10:41:34.898 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:34.911+0000 7f606e12b640 10 monclient(hunting): picked mon.noname-a con 0x7f606814a700 addr v1:172.21.15.88:6789/0 2024-03-29T10:41:34.898 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:34.911+0000 7f606e12b640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60681deb70 con 0x7f606814a700 2024-03-29T10:41:34.898 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:34.911+0000 7f606e12b640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6068147c40 con 0x7f60680ac5a0 2024-03-29T10:41:34.898 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:34.911+0000 7f606e12b640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6068125660 con 0x7f60680b0770 2024-03-29T10:41:34.898 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:34.911+0000 7f606e12b640 10 monclient(hunting): _renew_subs 2024-03-29T10:41:34.898 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:34.911+0000 7f606e12b640 10 monclient(hunting): authenticate will time out at 817.147688s 2024-03-29T10:41:34.899 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:34.916+0000 7f6067fff640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f606814a700 0x7f6068146f30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:55871/0 (socket says 172.21.15.88:55871) 2024-03-29T10:41:34.899 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:34.916+0000 7f6067fff640 1 -- 172.21.15.88:0/2292777063 learned_addr learned my addr 172.21.15.88:0/2292777063 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-29T10:41:34.900 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:34.916+0000 7f605ffff640 10 client.?.objecter ms_handle_connect 0x7f606814a700 2024-03-29T10:41:34.900 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:34.916+0000 7f605ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:41:34.900 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:34.917+0000 7f605ffff640 10 client.?.objecter ms_handle_connect 0x7f60680ac5a0 2024-03-29T10:41:34.900 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:34.917+0000 7f605ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:41:34.902 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:34.918+0000 7f605ffff640 1 -- 172.21.15.88:0/2292777063 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 356884495 0 0) 0x7f60681deb70 con 0x7f606814a700 2024-03-29T10:41:34.902 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:34.918+0000 7f605ffff640 10 monclient(hunting): _negotiate 2024-03-29T10:41:34.903 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:34.918+0000 7f605ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:41:34.903 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:34.918+0000 7f605ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:41:34.903 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:34.918+0000 7f605ffff640 10 monclient(hunting): my global_id is 4269 2024-03-29T10:41:34.903 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:34.918+0000 7f605ffff640 1 -- 172.21.15.88:0/2292777063 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6048004eb0 con 0x7f606814a700 2024-03-29T10:41:34.903 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:34.919+0000 7f605ffff640 1 -- 172.21.15.88:0/2292777063 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 948035213 0 0) 0x7f6068147c40 con 0x7f60680ac5a0 2024-03-29T10:41:34.903 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:34.919+0000 7f605ffff640 10 monclient(hunting): _negotiate 2024-03-29T10:41:34.903 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:34.919+0000 7f605ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:41:34.903 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:34.919+0000 7f605ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:41:34.903 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:34.919+0000 7f605ffff640 10 monclient(hunting): my global_id is 4255 2024-03-29T10:41:34.903 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:34.919+0000 7f605ffff640 1 -- 172.21.15.88:0/2292777063 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f60681deb70 con 0x7f60680ac5a0 2024-03-29T10:41:34.905 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:34.919+0000 7f605ffff640 1 -- 172.21.15.88:0/2292777063 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3035468960 0 0) 0x7f60681deb70 con 0x7f60680ac5a0 2024-03-29T10:41:34.905 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:34.919+0000 7f605ffff640 1 -- 172.21.15.88:0/2292777063 --> v1:172.21.15.196:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6068147c40 con 0x7f60680ac5a0 2024-03-29T10:41:34.905 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:34.919+0000 7f605ffff640 1 -- 172.21.15.88:0/2292777063 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f6060001870 con 0x7f60680ac5a0 2024-03-29T10:41:34.905 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:34.919+0000 7f605ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:34.905 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:34.919+0000 7f605ffff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:41:34.905 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:34.919+0000 7f605ffff640 10 monclient(hunting): dump: 2024-03-29T10:41:34.905 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:41:34.905 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:41:34.906 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:41:34.906 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:41:34.906 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:34.906 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:41:34.906 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:41:34.906 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:41:34.906 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:41:34.906 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:41:34.906 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:34.920+0000 7f605ffff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:41:34.906 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:34.920+0000 7f605ffff640 1 -- 172.21.15.88:0/2292777063 <== mon.1 v1:172.21.15.196:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 539925541 0 0) 0x7f6068147c40 con 0x7f60680ac5a0 2024-03-29T10:41:34.906 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:34.920+0000 7f605ffff640 1 -- 172.21.15.88:0/2292777063 >> v1:172.21.15.88:6790/0 conn(0x7f60680b0770 legacy=0x7f60681df090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:34.906 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:34.920+0000 7f605ffff640 1 -- 172.21.15.88:0/2292777063 >> v1:172.21.15.88:6789/0 conn(0x7f606814a700 legacy=0x7f6068146f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:34.907 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:34.920+0000 7f605ffff640 10 monclient: _finish_hunting 0 2024-03-29T10:41:34.907 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:34.920+0000 7f605ffff640 1 monclient: found mon.b 2024-03-29T10:41:34.907 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:34.920+0000 7f605ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:41:34.907 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:34.920+0000 7f605ffff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T10:41:34.907 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:34.920+0000 7f605ffff640 1 -- 172.21.15.88:0/2292777063 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60681df7a0 con 0x7f60680ac5a0 2024-03-29T10:41:34.907 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:34.920+0000 7f605ffff640 10 monclient: _finish_auth 0 2024-03-29T10:41:34.907 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:34.920+0000 7f605ffff640 10 monclient: _check_auth_tickets 2024-03-29T10:41:34.907 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:34.920+0000 7f605ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:41:34.907 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:34.920+0000 7f606e12b640 5 monclient: authenticate success, global_id 4255 2024-03-29T10:41:34.907 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:34.920+0000 7f606e12b640 10 monclient: _renew_subs 2024-03-29T10:41:34.907 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:34.920+0000 7f606e12b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T10:41:34.907 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:34.921+0000 7f606e12b640 1 -- 172.21.15.88:0/2292777063 --> v1:172.21.15.196:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60681dfa50 con 0x7f60680ac5a0 2024-03-29T10:41:34.907 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:34.921+0000 7f605ffff640 1 -- 172.21.15.88:0/2292777063 <== mon.1 v1:172.21.15.196:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f60600035f0 con 0x7f60680ac5a0 2024-03-29T10:41:34.907 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:34.921+0000 7f605ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:41:34.907 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:34.921+0000 7f605ffff640 1 -- 172.21.15.88:0/2292777063 <== mon.1 v1:172.21.15.196:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f6060001d80 con 0x7f60680ac5a0 2024-03-29T10:41:34.908 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:34.921+0000 7f605ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:34.908 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:34.921+0000 7f605ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:41:34.908 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:34.921+0000 7f605ffff640 10 monclient: dump: 2024-03-29T10:41:34.908 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:41:34.908 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:41:34.908 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:41:34.909 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:41:34.909 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:34.909 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:41:34.910 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:41:34.910 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:41:34.910 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:41:34.910 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:41:34.910 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:34.921+0000 7f606e12b640 10 client.4255.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:41:34.910 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:34.921+0000 7f606e12b640 10 monclient: _renew_subs 2024-03-29T10:41:34.910 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:34.921+0000 7f606e12b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T10:41:34.910 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:34.921+0000 7f606e12b640 1 -- 172.21.15.88:0/2292777063 --> v1:172.21.15.196:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f60681292c0 con 0x7f60680ac5a0 2024-03-29T10:41:34.910 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:34.921+0000 7f606e12b640 1 librados: init done 2024-03-29T10:41:34.910 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:34.921+0000 7f6065ffb640 4 set_mon_vals no callback set 2024-03-29T10:41:34.911 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:34.922+0000 7f606e12b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:41:34.912 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:34.923+0000 7f606e12b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:41:34.912 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:34.923+0000 7f606e12b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T10:41:34.912 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:34.923+0000 7f606e12b640 1 -- 172.21.15.88:0/2292777063 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f604c005080 con 0x7f60680ac5a0 2024-03-29T10:41:34.912 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:34.924+0000 7f605ffff640 1 -- 172.21.15.88:0/2292777063 <== mon.1 v1:172.21.15.196:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4235629220 0 0) 0x7f6060019770 con 0x7f60680ac5a0 2024-03-29T10:41:34.912 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:34.925+0000 7f605ffff640 1 -- 172.21.15.88:0/2292777063 <== mon.1 v1:172.21.15.196:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4257673582 0 0) 0x7f60600837e0 con 0x7f60680ac5a0 2024-03-29T10:41:34.912 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:34.925+0000 7f605ffff640 10 client.4255.objecter ms_dispatch 0x7f6068000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:41:34.912 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:34.925+0000 7f605ffff640 3 client.4255.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:41:34.912 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:34.925+0000 7f605ffff640 3 client.4255.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:41:34.912 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:34.925+0000 7f605ffff640 20 client.4255.objecter dump_active .. 0 homeless 2024-03-29T10:41:34.912 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:34.926+0000 7f6066ffd640 10 monclient: get_auth_request con 0x7f604806dda0 auth_method 0 2024-03-29T10:41:34.912 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:34.927+0000 7f605ffff640 10 client.4255.objecter ms_handle_connect 0x7f604806dda0 2024-03-29T10:41:34.915 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:34.932+0000 7f605ffff640 1 -- 172.21.15.88:0/2292777063 <== mon.1 v1:172.21.15.196:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f606004eef0 con 0x7f60680ac5a0 2024-03-29T10:41:34.915 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:34.932+0000 7f605ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:41:34.915 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:34.932+0000 7f605ffff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:41:35.235 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:35.251+0000 7f606e12b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:41:35.235 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:35.251+0000 7f606e12b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:41:35.235 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:35.251+0000 7f606e12b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T10:41:35.235 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:35.251+0000 7f606e12b640 1 -- 172.21.15.88:0/2292777063 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f604c005330 con 0x7f60680ac5a0 2024-03-29T10:41:35.239 INFO:teuthology.misc.health.smithi088.stdout: 2024-03-29T10:41:35.239 INFO:teuthology.misc.health.smithi088.stdout:{"epoch":10,"fsid":"869f5188-f92b-42dd-a46d-8c646e0388a9","created":"2024-03-29T10:40:40.285087+0000","modified":"2024-03-29T10:40:42.536230+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:40:42.536230+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a26c0c1c-8de0-438c-a3f6-3df10ea61574","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5de24915-8dec-49b6-8b16-f3170197c591","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ac054af3-d589-4565-9644-e6af75975c53","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3ec4f045-c714-4a22-bd1a-e07330b45ab3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"cddb9867-8e30-4951-9b13-3d1e78b9c45a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dc6d9911-785d-432f-8b53-56a1ccab302f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"77c88437-b587-496e-8b4d-1824ea8b5305","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ce963d96-5a8e-44c3-99de-c0dbebe234ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:41:35.239 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:35.253+0000 7f605ffff640 1 -- 172.21.15.88:0/2292777063 <== mon.1 v1:172.21.15.196:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 344506374) 0x7f6060050ff0 con 0x7f60680ac5a0 2024-03-29T10:41:35.240 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:35.253+0000 7f605ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:41:35.240 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:35.253+0000 7f605ffff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:41:35.240 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:35.256+0000 7f606e12b640 10 librados: watch_flush enter 2024-03-29T10:41:35.241 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:35.257+0000 7f606e12b640 10 librados: watch_flush exit 2024-03-29T10:41:35.241 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:35.257+0000 7f606e12b640 20 client.4255.objecter shutdown clearing up homeless session... 2024-03-29T10:41:35.241 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:35.257+0000 7f606e12b640 10 client.4255.objecter successfully canceled tick 2024-03-29T10:41:35.241 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:35.258+0000 7f606e12b640 1 -- 172.21.15.88:0/2292777063 >> v1:172.21.15.88:6816/1752861349 conn(0x7f604806dda0 legacy=0x7f6048070260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:35.241 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:35.258+0000 7f606e12b640 10 monclient: shutdown 2024-03-29T10:41:35.242 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:35.258+0000 7f606e12b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:41:35.242 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:35.258+0000 7f606e12b640 1 -- 172.21.15.88:0/2292777063 >> v1:172.21.15.196:6789/0 conn(0x7f60680ac5a0 legacy=0x7f60681281c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:35.243 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:35.259+0000 7f606e12b640 1 -- 172.21.15.88:0/2292777063 shutdown_connections 2024-03-29T10:41:35.243 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:35.259+0000 7f606e12b640 1 -- 172.21.15.88:0/2292777063 >> 172.21.15.88:0/2292777063 conn(0x7f606806b660 msgr2=0x7f606806ba40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:41:35.243 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:35.260+0000 7f606e12b640 1 -- 172.21.15.88:0/2292777063 shutdown_connections 2024-03-29T10:41:35.243 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:35.260+0000 7f606e12b640 1 -- 172.21.15.88:0/2292777063 wait complete. 2024-03-29T10:41:35.243 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:35.260+0000 7f606e12b640 1 librados: shutdown 2024-03-29T10:41:35.263 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:41:42.069 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:41:42.230 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:42.237+0000 7f8b89197640 10 monclient: get_monmap_and_config 2024-03-29T10:41:42.231 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:42.237+0000 7f8b89197640 10 monclient: build_initial_monmap 2024-03-29T10:41:42.231 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:42.237+0000 7f8b89197640 10 monclient: monmap: 2024-03-29T10:41:42.231 INFO:teuthology.misc.health.smithi088.stderr:epoch 0 2024-03-29T10:41:42.231 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:41:42.231 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:41:42.238986+0000 2024-03-29T10:41:42.231 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:41:42.238986+0000 2024-03-29T10:41:42.231 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 0 (unknown) 2024-03-29T10:41:42.232 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:41:42.232 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-03-29T10:41:42.232 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-29T10:41:42.232 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-b 2024-03-29T10:41:42.232 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:41:42.233 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:42.238+0000 7f8b89197640 1 Processor -- start 2024-03-29T10:41:42.233 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:42.238+0000 7f8b89197640 1 -- start start 2024-03-29T10:41:42.233 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:42.238+0000 7f8b89197640 10 monclient: init 2024-03-29T10:41:42.233 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:42.241+0000 7f8b89197640 10 monclient: _reopen_session rank -1 2024-03-29T10:41:42.233 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:42.241+0000 7f8b89197640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:41:42.233 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:42.241+0000 7f8b89197640 10 monclient(hunting): picked mon.noname-c con 0x7f8b840ad6c0 addr v1:172.21.15.196:6789/0 2024-03-29T10:41:42.233 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:42.241+0000 7f8b89197640 10 monclient(hunting): picked mon.noname-b con 0x7f8b840af6c0 addr v1:172.21.15.88:6790/0 2024-03-29T10:41:42.233 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:42.241+0000 7f8b89197640 10 monclient(hunting): picked mon.noname-a con 0x7f8b8414fdc0 addr v1:172.21.15.88:6789/0 2024-03-29T10:41:42.233 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:42.241+0000 7f8b89197640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b840a94e0 con 0x7f8b8414fdc0 2024-03-29T10:41:42.233 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:42.241+0000 7f8b89197640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b840af3a0 con 0x7f8b840ad6c0 2024-03-29T10:41:42.233 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:42.241+0000 7f8b89197640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b840afb60 con 0x7f8b840af6c0 2024-03-29T10:41:42.233 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:42.241+0000 7f8b89197640 10 monclient(hunting): _renew_subs 2024-03-29T10:41:42.233 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:42.241+0000 7f8b89197640 10 monclient(hunting): authenticate will time out at 824.477477s 2024-03-29T10:41:42.233 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:42.245+0000 7f8b8359e640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f8b8414fdc0 0x7f8b8415a8e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:40271/0 (socket says 172.21.15.88:40271) 2024-03-29T10:41:42.234 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:42.245+0000 7f8b8359e640 1 -- 172.21.15.88:0/1573073388 learned_addr learned my addr 172.21.15.88:0/1573073388 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-29T10:41:42.236 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:42.250+0000 7f8b81d9b640 1 -- 172.21.15.88:0/1573073388 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2663943880 0 0) 0x7f8b840a94e0 con 0x7f8b8414fdc0 2024-03-29T10:41:42.236 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:42.250+0000 7f8b81d9b640 10 monclient(hunting): _negotiate 2024-03-29T10:41:42.236 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:42.250+0000 7f8b81d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:41:42.236 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:42.250+0000 7f8b81d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:41:42.236 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:42.250+0000 7f8b81d9b640 10 monclient(hunting): my global_id is 4272 2024-03-29T10:41:42.236 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:42.250+0000 7f8b81d9b640 1 -- 172.21.15.88:0/1573073388 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8b6c003500 con 0x7f8b8414fdc0 2024-03-29T10:41:42.236 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:42.250+0000 7f8b81d9b640 1 -- 172.21.15.88:0/1573073388 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 385332533 0 0) 0x7f8b840af3a0 con 0x7f8b840ad6c0 2024-03-29T10:41:42.236 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:42.250+0000 7f8b81d9b640 10 monclient(hunting): _negotiate 2024-03-29T10:41:42.236 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:42.250+0000 7f8b81d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:41:42.236 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:42.250+0000 7f8b81d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:41:42.236 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:42.250+0000 7f8b81d9b640 10 monclient(hunting): my global_id is 4258 2024-03-29T10:41:42.236 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:42.250+0000 7f8b81d9b640 1 -- 172.21.15.88:0/1573073388 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8b840a94e0 con 0x7f8b840ad6c0 2024-03-29T10:41:42.236 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:42.251+0000 7f8b81d9b640 1 -- 172.21.15.88:0/1573073388 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1511602547 0 0) 0x7f8b840afb60 con 0x7f8b840af6c0 2024-03-29T10:41:42.236 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:42.251+0000 7f8b81d9b640 10 monclient(hunting): _negotiate 2024-03-29T10:41:42.237 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:42.251+0000 7f8b81d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:41:42.237 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:42.251+0000 7f8b81d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:41:42.237 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:42.251+0000 7f8b81d9b640 10 monclient(hunting): my global_id is 4262 2024-03-29T10:41:42.237 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:42.251+0000 7f8b81d9b640 1 -- 172.21.15.88:0/1573073388 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8b840af3a0 con 0x7f8b840af6c0 2024-03-29T10:41:42.237 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:42.251+0000 7f8b81d9b640 1 -- 172.21.15.88:0/1573073388 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 141546110 0 0) 0x7f8b840af3a0 con 0x7f8b840af6c0 2024-03-29T10:41:42.237 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:42.251+0000 7f8b81d9b640 1 -- 172.21.15.88:0/1573073388 >> v1:172.21.15.196:6789/0 conn(0x7f8b840ad6c0 legacy=0x7f8b8414f6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:42.237 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:42.251+0000 7f8b81d9b640 1 -- 172.21.15.88:0/1573073388 >> v1:172.21.15.88:6789/0 conn(0x7f8b8414fdc0 legacy=0x7f8b8415a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:42.237 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:42.251+0000 7f8b81d9b640 10 monclient: _finish_hunting 0 2024-03-29T10:41:42.237 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:42.251+0000 7f8b81d9b640 1 monclient: found mon.noname-b 2024-03-29T10:41:42.237 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:42.251+0000 7f8b81d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:41:42.237 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:42.251+0000 7f8b81d9b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.88:6790/0 2024-03-29T10:41:42.237 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:42.251+0000 7f8b81d9b640 1 -- 172.21.15.88:0/1573073388 --> v1:172.21.15.88:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b840aab80 con 0x7f8b840af6c0 2024-03-29T10:41:42.237 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:42.251+0000 7f8b81d9b640 10 monclient: _finish_auth 0 2024-03-29T10:41:42.237 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:42.251+0000 7f8b81d9b640 10 monclient: _check_auth_tickets 2024-03-29T10:41:42.237 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:42.251+0000 7f8b81d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:41:42.238 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:42.251+0000 7f8b89197640 5 monclient: authenticate success, global_id 4262 2024-03-29T10:41:42.238 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:42.251+0000 7f8b89197640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:41:42.238 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:42.252+0000 7f8b81d9b640 1 -- 172.21.15.88:0/1573073388 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f8b74001630 con 0x7f8b840af6c0 2024-03-29T10:41:42.238 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:42.252+0000 7f8b81d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:42.238 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:42.252+0000 7f8b81d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:41:42.238 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:42.252+0000 7f8b81d9b640 10 monclient: dump: 2024-03-29T10:41:42.238 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:41:42.238 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:41:42.238 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:41:42.238 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:41:42.238 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:42.238 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:41:42.238 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:41:42.238 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:41:42.238 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:41:42.238 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:41:42.238 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:42.252+0000 7f8b89197640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:41:42.240 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:42.252+0000 7f8b81d9b640 1 -- 172.21.15.88:0/1573073388 <== mon.2 v1:172.21.15.88:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8b74001790 con 0x7f8b840af6c0 2024-03-29T10:41:42.240 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:42.252+0000 7f8b81d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:41:42.240 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:42.252+0000 7f8b89197640 10 monclient: get_monmap_and_config success 2024-03-29T10:41:42.240 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:42.252+0000 7f8b89197640 4 set_mon_vals no callback set 2024-03-29T10:41:42.240 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:42.253+0000 7f8b81d9b640 1 -- 172.21.15.88:0/1573073388 <== mon.2 v1:172.21.15.88:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f8b74001be0 con 0x7f8b840af6c0 2024-03-29T10:41:42.240 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:42.253+0000 7f8b81d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:42.240 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:42.253+0000 7f8b81d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:41:42.240 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:42.253+0000 7f8b81d9b640 10 monclient: dump: 2024-03-29T10:41:42.240 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:41:42.240 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:41:42.240 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:41:42.240 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:41:42.241 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:42.241 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:41:42.241 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:41:42.241 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:41:42.241 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:41:42.241 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:41:42.241 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:42.253+0000 7f8b89197640 10 monclient: shutdown 2024-03-29T10:41:42.241 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:42.253+0000 7f8b89197640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:41:42.241 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:42.253+0000 7f8b89197640 1 -- 172.21.15.88:0/1573073388 >> v1:172.21.15.88:6790/0 conn(0x7f8b840af6c0 legacy=0x7f8b84152980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:42.241 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:42.253+0000 7f8b89197640 1 -- 172.21.15.88:0/1573073388 shutdown_connections 2024-03-29T10:41:42.241 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:42.253+0000 7f8b89197640 1 -- 172.21.15.88:0/1573073388 >> 172.21.15.88:0/1573073388 conn(0x7f8b840b35e0 msgr2=0x7f8b840b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:41:42.241 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:42.254+0000 7f8b89197640 1 -- 172.21.15.88:0/1573073388 shutdown_connections 2024-03-29T10:41:42.241 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:42.254+0000 7f8b89197640 1 -- 172.21.15.88:0/1573073388 wait complete. 2024-03-29T10:41:42.241 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:42.255+0000 7f8b89197640 10 monclient: build_initial_monmap 2024-03-29T10:41:42.241 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:42.255+0000 7f8b89197640 10 monclient: monmap: 2024-03-29T10:41:42.241 INFO:teuthology.misc.health.smithi088.stderr:epoch 0 2024-03-29T10:41:42.241 INFO:teuthology.misc.health.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:41:42.242 INFO:teuthology.misc.health.smithi088.stderr:last_changed 0.000000 2024-03-29T10:41:42.242 INFO:teuthology.misc.health.smithi088.stderr:created 0.000000 2024-03-29T10:41:42.242 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 0 (unknown) 2024-03-29T10:41:42.242 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:41:42.242 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-03-29T10:41:42.242 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-29T10:41:42.242 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-c 2024-03-29T10:41:42.242 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:41:42.242 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:42.255+0000 7f8b89197640 1 librados: starting msgr at 2024-03-29T10:41:42.242 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:42.255+0000 7f8b89197640 1 librados: starting objecter 2024-03-29T10:41:42.242 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:42.255+0000 7f8b89197640 1 Processor -- start 2024-03-29T10:41:42.243 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:42.255+0000 7f8b89197640 1 -- start start 2024-03-29T10:41:42.243 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:42.255+0000 7f8b89197640 1 librados: setting wanted keys 2024-03-29T10:41:42.243 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:42.255+0000 7f8b89197640 1 librados: calling monclient init 2024-03-29T10:41:42.243 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:42.255+0000 7f8b89197640 10 monclient: init 2024-03-29T10:41:42.243 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:42.256+0000 7f8b89197640 10 monclient: _reopen_session rank -1 2024-03-29T10:41:42.243 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:42.256+0000 7f8b89197640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:41:42.243 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:42.256+0000 7f8b89197640 10 monclient(hunting): picked mon.noname-b con 0x7f8b840ad6c0 addr v1:172.21.15.196:6789/0 2024-03-29T10:41:42.243 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:42.256+0000 7f8b89197640 10 monclient(hunting): picked mon.noname-a con 0x7f8b8414fdc0 addr v1:172.21.15.88:6789/0 2024-03-29T10:41:42.243 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:42.256+0000 7f8b89197640 10 monclient(hunting): picked mon.noname-c con 0x7f8b8413e400 addr v1:172.21.15.88:6790/0 2024-03-29T10:41:42.243 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:42.256+0000 7f8b89197640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b8413e250 con 0x7f8b8414fdc0 2024-03-29T10:41:42.243 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:42.256+0000 7f8b89197640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b8413ea20 con 0x7f8b840ad6c0 2024-03-29T10:41:42.243 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:42.256+0000 7f8b89197640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b8413eeb0 con 0x7f8b8413e400 2024-03-29T10:41:42.243 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:42.256+0000 7f8b89197640 10 monclient(hunting): _renew_subs 2024-03-29T10:41:42.243 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:42.256+0000 7f8b89197640 10 monclient(hunting): authenticate will time out at 824.492686s 2024-03-29T10:41:42.245 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:42.261+0000 7f8b8359e640 1 --1- >> v1:172.21.15.88:6790/0 conn(0x7f8b8413e400 0x7f8b840705b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6790/0 says I am v1:172.21.15.88:57919/0 (socket says 172.21.15.88:57919) 2024-03-29T10:41:42.245 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:42.261+0000 7f8b8359e640 1 -- 172.21.15.88:0/806862474 learned_addr learned my addr 172.21.15.88:0/806862474 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-29T10:41:42.247 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:42.263+0000 7f8b6b7fe640 10 client.?.objecter ms_handle_connect 0x7f8b8413e400 2024-03-29T10:41:42.247 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:42.263+0000 7f8b6b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:41:42.247 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:42.263+0000 7f8b6b7fe640 1 -- 172.21.15.88:0/806862474 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 659466450 0 0) 0x7f8b8413eeb0 con 0x7f8b8413e400 2024-03-29T10:41:42.247 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:42.263+0000 7f8b6b7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:41:42.247 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:42.263+0000 7f8b6b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:41:42.247 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:42.263+0000 7f8b6b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:41:42.248 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:42.263+0000 7f8b6b7fe640 10 monclient(hunting): my global_id is 4265 2024-03-29T10:41:42.248 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:42.263+0000 7f8b6b7fe640 1 -- 172.21.15.88:0/806862474 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8b70004eb0 con 0x7f8b8413e400 2024-03-29T10:41:42.248 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:42.263+0000 7f8b6b7fe640 10 client.?.objecter ms_handle_connect 0x7f8b840ad6c0 2024-03-29T10:41:42.248 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:42.263+0000 7f8b6b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:41:42.248 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:42.263+0000 7f8b6b7fe640 1 -- 172.21.15.88:0/806862474 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 610070260 0 0) 0x7f8b8413ea20 con 0x7f8b840ad6c0 2024-03-29T10:41:42.248 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:42.263+0000 7f8b6b7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:41:42.248 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:42.264+0000 7f8b6b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:41:42.248 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:42.264+0000 7f8b6b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:41:42.248 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:42.264+0000 7f8b6b7fe640 10 monclient(hunting): my global_id is 4261 2024-03-29T10:41:42.248 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:42.264+0000 7f8b6b7fe640 1 -- 172.21.15.88:0/806862474 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8b8413eeb0 con 0x7f8b840ad6c0 2024-03-29T10:41:42.251 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:42.265+0000 7f8b6b7fe640 1 -- 172.21.15.88:0/806862474 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3305983658 0 0) 0x7f8b70004eb0 con 0x7f8b8413e400 2024-03-29T10:41:42.251 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:42.265+0000 7f8b6b7fe640 1 -- 172.21.15.88:0/806862474 --> v1:172.21.15.88:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8b8413ea20 con 0x7f8b8413e400 2024-03-29T10:41:42.251 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:42.265+0000 7f8b6b7fe640 1 -- 172.21.15.88:0/806862474 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f8b78001810 con 0x7f8b8413e400 2024-03-29T10:41:42.251 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:42.265+0000 7f8b6b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:42.251 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:42.265+0000 7f8b6b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:41:42.251 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:42.265+0000 7f8b6b7fe640 10 monclient(hunting): dump: 2024-03-29T10:41:42.251 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:41:42.251 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:41:42.251 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:41:42.252 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:41:42.252 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:42.252 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:41:42.252 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:41:42.252 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:41:42.252 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:41:42.252 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:41:42.252 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:42.265+0000 7f8b6b7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:41:42.252 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:42.265+0000 7f8b6b7fe640 10 client.?.objecter ms_handle_connect 0x7f8b8414fdc0 2024-03-29T10:41:42.252 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:42.265+0000 7f8b6b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:41:42.252 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:42.265+0000 7f8b6b7fe640 1 -- 172.21.15.88:0/806862474 <== mon.2 v1:172.21.15.88:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2457933719 0 0) 0x7f8b8413ea20 con 0x7f8b8413e400 2024-03-29T10:41:42.252 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:42.265+0000 7f8b6b7fe640 1 -- 172.21.15.88:0/806862474 >> v1:172.21.15.196:6789/0 conn(0x7f8b840ad6c0 legacy=0x7f8b8413db40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:42.252 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:42.265+0000 7f8b6b7fe640 1 -- 172.21.15.88:0/806862474 >> v1:172.21.15.88:6789/0 conn(0x7f8b8414fdc0 legacy=0x7f8b8408a6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:42.252 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:42.265+0000 7f8b6b7fe640 10 monclient: _finish_hunting 0 2024-03-29T10:41:42.252 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:42.266+0000 7f8b6b7fe640 1 monclient: found mon.c 2024-03-29T10:41:42.253 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:42.266+0000 7f8b6b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:41:42.253 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:42.266+0000 7f8b6b7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-03-29T10:41:42.253 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:42.266+0000 7f8b6b7fe640 1 -- 172.21.15.88:0/806862474 --> v1:172.21.15.88:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b8413fc50 con 0x7f8b8413e400 2024-03-29T10:41:42.253 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:42.266+0000 7f8b6b7fe640 10 monclient: _finish_auth 0 2024-03-29T10:41:42.253 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:42.266+0000 7f8b6b7fe640 10 monclient: _check_auth_tickets 2024-03-29T10:41:42.253 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:42.266+0000 7f8b6b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:41:42.253 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:42.266+0000 7f8b89197640 5 monclient: authenticate success, global_id 4265 2024-03-29T10:41:42.253 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:42.266+0000 7f8b89197640 10 monclient: _renew_subs 2024-03-29T10:41:42.253 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:42.266+0000 7f8b89197640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-03-29T10:41:42.253 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:42.266+0000 7f8b89197640 1 -- 172.21.15.88:0/806862474 --> v1:172.21.15.88:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b8408ae00 con 0x7f8b8413e400 2024-03-29T10:41:42.253 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:42.266+0000 7f8b89197640 10 client.4265.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:41:42.253 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:42.266+0000 7f8b89197640 10 monclient: _renew_subs 2024-03-29T10:41:42.253 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:42.266+0000 7f8b89197640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-03-29T10:41:42.253 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:42.266+0000 7f8b89197640 1 -- 172.21.15.88:0/806862474 --> v1:172.21.15.88:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b84070cc0 con 0x7f8b8413e400 2024-03-29T10:41:42.253 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:42.266+0000 7f8b89197640 1 librados: init done 2024-03-29T10:41:42.254 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:42.267+0000 7f8b89197640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:41:42.254 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:42.267+0000 7f8b89197640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:41:42.254 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:42.267+0000 7f8b6b7fe640 1 -- 172.21.15.88:0/806862474 <== mon.2 v1:172.21.15.88:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8b78001d00 con 0x7f8b8413e400 2024-03-29T10:41:42.254 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:42.267+0000 7f8b89197640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-03-29T10:41:42.254 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:42.267+0000 7f8b89197640 1 -- 172.21.15.88:0/806862474 --> v1:172.21.15.88:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b84075f10 con 0x7f8b8413e400 2024-03-29T10:41:42.254 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:42.267+0000 7f8b6b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:41:42.255 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:42.267+0000 7f8b8159a640 4 set_mon_vals no callback set 2024-03-29T10:41:42.255 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:42.268+0000 7f8b6b7fe640 1 -- 172.21.15.88:0/806862474 <== mon.2 v1:172.21.15.88:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f8b78004980 con 0x7f8b8413e400 2024-03-29T10:41:42.256 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:42.268+0000 7f8b6b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:42.256 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:42.268+0000 7f8b6b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:41:42.256 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:42.268+0000 7f8b6b7fe640 10 monclient: dump: 2024-03-29T10:41:42.256 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:41:42.256 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:41:42.256 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:41:42.256 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:41:42.256 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:42.256 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:41:42.256 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:41:42.256 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:41:42.256 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:41:42.256 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:41:42.256 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:42.269+0000 7f8b6b7fe640 1 -- 172.21.15.88:0/806862474 <== mon.2 v1:172.21.15.88:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4235629220 0 0) 0x7f8b78004ae0 con 0x7f8b8413e400 2024-03-29T10:41:42.257 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:42.270+0000 7f8b6b7fe640 1 -- 172.21.15.88:0/806862474 <== mon.2 v1:172.21.15.88:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4257673582 0 0) 0x7f8b78084a10 con 0x7f8b8413e400 2024-03-29T10:41:42.257 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:42.270+0000 7f8b6b7fe640 10 client.4265.objecter ms_dispatch 0x7f8b84000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:41:42.257 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:42.270+0000 7f8b6b7fe640 3 client.4265.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:41:42.257 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:42.270+0000 7f8b6b7fe640 3 client.4265.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:41:42.257 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:42.271+0000 7f8b6b7fe640 20 client.4265.objecter dump_active .. 0 homeless 2024-03-29T10:41:42.257 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:42.271+0000 7f8b82d9d640 10 monclient: get_auth_request con 0x7f8b7006dd60 auth_method 0 2024-03-29T10:41:42.257 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:42.271+0000 7f8b6b7fe640 10 client.4265.objecter ms_handle_connect 0x7f8b7006dd60 2024-03-29T10:41:42.263 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:42.279+0000 7f8b6b7fe640 1 -- 172.21.15.88:0/806862474 <== mon.2 v1:172.21.15.88:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8b78050120 con 0x7f8b8413e400 2024-03-29T10:41:42.263 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:42.280+0000 7f8b6b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:41:42.263 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:42.280+0000 7f8b6b7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:41:42.592 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:42.608+0000 7f8b89197640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:41:42.592 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:42.608+0000 7f8b89197640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:41:42.592 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:42.609+0000 7f8b89197640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-03-29T10:41:42.592 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:42.609+0000 7f8b89197640 1 -- 172.21.15.88:0/806862474 --> v1:172.21.15.88:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8b840ae770 con 0x7f8b8413e400 2024-03-29T10:41:42.593 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:42.609+0000 7f8b6b7fe640 1 -- 172.21.15.88:0/806862474 <== mon.2 v1:172.21.15.88:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 344506374) 0x7f8b78052220 con 0x7f8b8413e400 2024-03-29T10:41:42.593 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:42.609+0000 7f8b6b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:41:42.593 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:42.609+0000 7f8b6b7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:41:42.597 INFO:teuthology.misc.health.smithi088.stdout: 2024-03-29T10:41:42.597 INFO:teuthology.misc.health.smithi088.stdout:{"epoch":10,"fsid":"869f5188-f92b-42dd-a46d-8c646e0388a9","created":"2024-03-29T10:40:40.285087+0000","modified":"2024-03-29T10:40:42.536230+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:40:42.536230+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a26c0c1c-8de0-438c-a3f6-3df10ea61574","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5de24915-8dec-49b6-8b16-f3170197c591","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ac054af3-d589-4565-9644-e6af75975c53","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3ec4f045-c714-4a22-bd1a-e07330b45ab3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"cddb9867-8e30-4951-9b13-3d1e78b9c45a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dc6d9911-785d-432f-8b53-56a1ccab302f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"77c88437-b587-496e-8b4d-1824ea8b5305","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ce963d96-5a8e-44c3-99de-c0dbebe234ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:41:42.600 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:42.615+0000 7f8b89197640 10 librados: watch_flush enter 2024-03-29T10:41:42.600 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:42.615+0000 7f8b89197640 10 librados: watch_flush exit 2024-03-29T10:41:42.600 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:42.615+0000 7f8b89197640 20 client.4265.objecter shutdown clearing up homeless session... 2024-03-29T10:41:42.600 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:42.615+0000 7f8b89197640 10 client.4265.objecter successfully canceled tick 2024-03-29T10:41:42.600 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:42.616+0000 7f8b89197640 1 -- 172.21.15.88:0/806862474 >> v1:172.21.15.88:6816/1752861349 conn(0x7f8b7006dd60 legacy=0x7f8b70070220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:42.600 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:42.616+0000 7f8b89197640 10 monclient: shutdown 2024-03-29T10:41:42.600 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:42.616+0000 7f8b89197640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:41:42.600 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:42.616+0000 7f8b89197640 1 -- 172.21.15.88:0/806862474 >> v1:172.21.15.88:6790/0 conn(0x7f8b8413e400 legacy=0x7f8b840705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:42.600 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:42.616+0000 7f8b89197640 1 -- 172.21.15.88:0/806862474 shutdown_connections 2024-03-29T10:41:42.600 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:42.616+0000 7f8b89197640 1 -- 172.21.15.88:0/806862474 >> 172.21.15.88:0/806862474 conn(0x7f8b840b35e0 msgr2=0x7f8b8414dbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:41:42.601 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:42.617+0000 7f8b89197640 1 -- 172.21.15.88:0/806862474 shutdown_connections 2024-03-29T10:41:42.602 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:42.619+0000 7f8b89197640 1 -- 172.21.15.88:0/806862474 wait complete. 2024-03-29T10:41:42.602 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:42.619+0000 7f8b89197640 1 librados: shutdown 2024-03-29T10:41:42.629 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:41:49.436 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:41:49.588 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:49.604+0000 7fac3a9f3640 10 monclient: get_monmap_and_config 2024-03-29T10:41:49.588 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:49.604+0000 7fac3a9f3640 10 monclient: build_initial_monmap 2024-03-29T10:41:49.588 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:49.604+0000 7fac3a9f3640 10 monclient: monmap: 2024-03-29T10:41:49.588 INFO:teuthology.misc.health.smithi088.stderr:epoch 0 2024-03-29T10:41:49.588 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:41:49.588 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:41:49.605198+0000 2024-03-29T10:41:49.588 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:41:49.605198+0000 2024-03-29T10:41:49.589 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 0 (unknown) 2024-03-29T10:41:49.589 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:41:49.589 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-03-29T10:41:49.589 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-29T10:41:49.589 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-b 2024-03-29T10:41:49.589 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:41:49.590 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:49.607+0000 7fac3a9f3640 1 Processor -- start 2024-03-29T10:41:49.593 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:49.607+0000 7fac3a9f3640 1 -- start start 2024-03-29T10:41:49.593 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:49.607+0000 7fac3a9f3640 10 monclient: init 2024-03-29T10:41:49.593 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:49.608+0000 7fac3a9f3640 10 monclient: _reopen_session rank -1 2024-03-29T10:41:49.593 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:49.608+0000 7fac3a9f3640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:41:49.593 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:49.608+0000 7fac3a9f3640 10 monclient(hunting): picked mon.noname-a con 0x7fac2c006a70 addr v1:172.21.15.88:6789/0 2024-03-29T10:41:49.593 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:49.608+0000 7fac3a9f3640 10 monclient(hunting): picked mon.noname-c con 0x7fac2c02b100 addr v1:172.21.15.196:6789/0 2024-03-29T10:41:49.593 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:49.608+0000 7fac3a9f3640 10 monclient(hunting): picked mon.noname-b con 0x7fac2c029c30 addr v1:172.21.15.88:6790/0 2024-03-29T10:41:49.593 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:49.608+0000 7fac3a9f3640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac2c02a270 con 0x7fac2c006a70 2024-03-29T10:41:49.594 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:49.608+0000 7fac3a9f3640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac2c02c440 con 0x7fac2c02b100 2024-03-29T10:41:49.594 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:49.608+0000 7fac3a9f3640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac2c0c9440 con 0x7fac2c029c30 2024-03-29T10:41:49.594 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:49.608+0000 7fac3a9f3640 10 monclient(hunting): _renew_subs 2024-03-29T10:41:49.594 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:49.608+0000 7fac3a9f3640 10 monclient(hunting): authenticate will time out at 831.844269s 2024-03-29T10:41:49.597 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:49.611+0000 7fac399f1640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fac2c006a70 0x7fac2c0c8d30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:36223/0 (socket says 172.21.15.88:36223) 2024-03-29T10:41:49.597 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:49.611+0000 7fac399f1640 1 -- 172.21.15.88:0/977434714 learned_addr learned my addr 172.21.15.88:0/977434714 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-29T10:41:49.597 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:49.612+0000 7fac389ef640 1 -- 172.21.15.88:0/977434714 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3894039812 0 0) 0x7fac2c02a270 con 0x7fac2c006a70 2024-03-29T10:41:49.597 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:49.612+0000 7fac389ef640 10 monclient(hunting): _negotiate 2024-03-29T10:41:49.597 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:49.612+0000 7fac389ef640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:41:49.597 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:49.612+0000 7fac389ef640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:41:49.597 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:49.612+0000 7fac389ef640 10 monclient(hunting): my global_id is 4275 2024-03-29T10:41:49.597 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:49.612+0000 7fac389ef640 1 -- 172.21.15.88:0/977434714 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fac20003520 con 0x7fac2c006a70 2024-03-29T10:41:49.597 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:49.612+0000 7fac389ef640 1 -- 172.21.15.88:0/977434714 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1215351816 0 0) 0x7fac2c0c9440 con 0x7fac2c029c30 2024-03-29T10:41:49.597 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:49.612+0000 7fac389ef640 10 monclient(hunting): _negotiate 2024-03-29T10:41:49.598 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:49.613+0000 7fac389ef640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:41:49.598 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:49.613+0000 7fac389ef640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:41:49.598 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:49.613+0000 7fac389ef640 10 monclient(hunting): my global_id is 4268 2024-03-29T10:41:49.598 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:49.613+0000 7fac389ef640 1 -- 172.21.15.88:0/977434714 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fac2c02a270 con 0x7fac2c029c30 2024-03-29T10:41:49.598 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:49.613+0000 7fac389ef640 1 -- 172.21.15.88:0/977434714 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3569033112 0 0) 0x7fac20003520 con 0x7fac2c006a70 2024-03-29T10:41:49.598 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:49.613+0000 7fac389ef640 1 -- 172.21.15.88:0/977434714 >> v1:172.21.15.88:6790/0 conn(0x7fac2c029c30 legacy=0x7fac2c0d74a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:49.598 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:49.613+0000 7fac389ef640 1 -- 172.21.15.88:0/977434714 >> v1:172.21.15.196:6789/0 conn(0x7fac2c02b100 legacy=0x7fac2c0cf100 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:41:49.600 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:49.616+0000 7fac389ef640 10 monclient: _finish_hunting 0 2024-03-29T10:41:49.600 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:49.616+0000 7fac389ef640 1 monclient: found mon.noname-a 2024-03-29T10:41:49.600 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:49.616+0000 7fac389ef640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:41:49.600 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:49.616+0000 7fac389ef640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.88:6789/0 2024-03-29T10:41:49.600 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:49.616+0000 7fac389ef640 1 -- 172.21.15.88:0/977434714 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac2c0c95d0 con 0x7fac2c006a70 2024-03-29T10:41:49.600 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:49.616+0000 7fac389ef640 10 monclient: _finish_auth 0 2024-03-29T10:41:49.601 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:49.616+0000 7fac389ef640 10 monclient: _check_auth_tickets 2024-03-29T10:41:49.601 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:49.616+0000 7fac389ef640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:41:49.601 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:49.616+0000 7fac389ef640 1 -- 172.21.15.88:0/977434714 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7fac3404f730 con 0x7fac2c006a70 2024-03-29T10:41:49.601 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:49.616+0000 7fac389ef640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:49.601 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:49.616+0000 7fac389ef640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:41:49.601 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:49.616+0000 7fac389ef640 10 monclient: dump: 2024-03-29T10:41:49.601 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:41:49.601 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:41:49.601 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:41:49.601 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:41:49.601 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:49.601 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:41:49.601 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:41:49.601 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:41:49.601 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:41:49.601 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:41:49.602 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:49.617+0000 7fac3a9f3640 5 monclient: authenticate success, global_id 4275 2024-03-29T10:41:49.602 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:49.618+0000 7fac3a9f3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:41:49.602 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:49.619+0000 7fac389ef640 1 -- 172.21.15.88:0/977434714 <== mon.0 v1:172.21.15.88:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fac34075f10 con 0x7fac2c006a70 2024-03-29T10:41:49.603 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:49.619+0000 7fac389ef640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:41:49.604 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:49.619+0000 7fac3a9f3640 10 monclient: get_monmap_and_config success 2024-03-29T10:41:49.604 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:49.619+0000 7fac3a9f3640 4 set_mon_vals no callback set 2024-03-29T10:41:49.604 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:49.620+0000 7fac3a9f3640 10 monclient: shutdown 2024-03-29T10:41:49.604 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:49.620+0000 7fac3a9f3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:41:49.604 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:49.620+0000 7fac3a9f3640 1 -- 172.21.15.88:0/977434714 >> v1:172.21.15.88:6789/0 conn(0x7fac2c006a70 legacy=0x7fac2c0c8d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:49.604 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:49.621+0000 7fac3a9f3640 1 -- 172.21.15.88:0/977434714 shutdown_connections 2024-03-29T10:41:49.604 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:49.621+0000 7fac3a9f3640 1 -- 172.21.15.88:0/977434714 >> 172.21.15.88:0/977434714 conn(0x7fac2c0058f0 msgr2=0x7fac2c005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:41:49.607 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:49.623+0000 7fac3a9f3640 1 -- 172.21.15.88:0/977434714 shutdown_connections 2024-03-29T10:41:49.608 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:49.625+0000 7fac3a9f3640 1 -- 172.21.15.88:0/977434714 wait complete. 2024-03-29T10:41:49.610 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:49.626+0000 7fac3a9f3640 10 monclient: build_initial_monmap 2024-03-29T10:41:49.610 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:49.626+0000 7fac3a9f3640 10 monclient: monmap: 2024-03-29T10:41:49.610 INFO:teuthology.misc.health.smithi088.stderr:epoch 0 2024-03-29T10:41:49.610 INFO:teuthology.misc.health.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:41:49.610 INFO:teuthology.misc.health.smithi088.stderr:last_changed 0.000000 2024-03-29T10:41:49.610 INFO:teuthology.misc.health.smithi088.stderr:created 0.000000 2024-03-29T10:41:49.610 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 0 (unknown) 2024-03-29T10:41:49.610 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:41:49.610 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-03-29T10:41:49.610 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-29T10:41:49.611 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-c 2024-03-29T10:41:49.611 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:41:49.611 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:49.626+0000 7fac3a9f3640 1 librados: starting msgr at 2024-03-29T10:41:49.611 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:49.626+0000 7fac3a9f3640 1 librados: starting objecter 2024-03-29T10:41:49.611 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:49.626+0000 7fac3a9f3640 1 Processor -- start 2024-03-29T10:41:49.611 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:49.627+0000 7fac3a9f3640 1 -- start start 2024-03-29T10:41:49.611 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:49.627+0000 7fac3a9f3640 1 librados: setting wanted keys 2024-03-29T10:41:49.611 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:49.627+0000 7fac3a9f3640 1 librados: calling monclient init 2024-03-29T10:41:49.611 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:49.627+0000 7fac3a9f3640 10 monclient: init 2024-03-29T10:41:49.611 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:49.627+0000 7fac3a9f3640 10 monclient: _reopen_session rank -1 2024-03-29T10:41:49.612 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:49.627+0000 7fac3a9f3640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:41:49.612 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:49.627+0000 7fac3a9f3640 10 monclient(hunting): picked mon.noname-c con 0x7fac2c006a70 addr v1:172.21.15.88:6790/0 2024-03-29T10:41:49.612 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:49.627+0000 7fac3a9f3640 10 monclient(hunting): picked mon.noname-b con 0x7fac2c029c30 addr v1:172.21.15.196:6789/0 2024-03-29T10:41:49.612 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:49.627+0000 7fac3a9f3640 10 monclient(hunting): picked mon.noname-a con 0x7fac2c02d890 addr v1:172.21.15.88:6789/0 2024-03-29T10:41:49.612 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:49.627+0000 7fac3a9f3640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac2c02ebf0 con 0x7fac2c02d890 2024-03-29T10:41:49.612 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:49.627+0000 7fac3a9f3640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac2c02ee60 con 0x7fac2c029c30 2024-03-29T10:41:49.612 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:49.627+0000 7fac3a9f3640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac2c0cccb0 con 0x7fac2c006a70 2024-03-29T10:41:49.612 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:49.627+0000 7fac3a9f3640 10 monclient(hunting): _renew_subs 2024-03-29T10:41:49.612 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:49.627+0000 7fac3a9f3640 10 monclient(hunting): authenticate will time out at 831.863748s 2024-03-29T10:41:49.618 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:49.635+0000 7fac399f1640 1 --1- >> v1:172.21.15.88:6790/0 conn(0x7fac2c006a70 0x7fac2c02cb60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6790/0 says I am v1:172.21.15.88:46123/0 (socket says 172.21.15.88:46123) 2024-03-29T10:41:49.618 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:49.635+0000 7fac399f1640 1 -- 172.21.15.88:0/517489519 learned_addr learned my addr 172.21.15.88:0/517489519 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-29T10:41:49.623 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:49.637+0000 7fac2a7fc640 10 client.?.objecter ms_handle_connect 0x7fac2c029c30 2024-03-29T10:41:49.623 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:49.637+0000 7fac2a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:41:49.623 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:49.637+0000 7fac2a7fc640 10 client.?.objecter ms_handle_connect 0x7fac2c006a70 2024-03-29T10:41:49.623 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:49.637+0000 7fac2a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:41:49.623 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:49.637+0000 7fac2a7fc640 10 client.?.objecter ms_handle_connect 0x7fac2c02d890 2024-03-29T10:41:49.623 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:49.637+0000 7fac2a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:41:49.624 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:49.637+0000 7fac2a7fc640 1 -- 172.21.15.88:0/517489519 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1846610328 0 0) 0x7fac2c0cccb0 con 0x7fac2c006a70 2024-03-29T10:41:49.624 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:49.637+0000 7fac2a7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:41:49.624 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:49.637+0000 7fac2a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:41:49.624 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:49.637+0000 7fac2a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:41:49.624 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:49.638+0000 7fac2a7fc640 10 monclient(hunting): my global_id is 4271 2024-03-29T10:41:49.624 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:49.638+0000 7fac2a7fc640 1 -- 172.21.15.88:0/517489519 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fac18004eb0 con 0x7fac2c006a70 2024-03-29T10:41:49.624 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:49.638+0000 7fac2a7fc640 1 -- 172.21.15.88:0/517489519 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 81830953 0 0) 0x7fac2c02ee60 con 0x7fac2c029c30 2024-03-29T10:41:49.624 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:49.638+0000 7fac2a7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:41:49.624 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:49.638+0000 7fac2a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:41:49.624 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:49.638+0000 7fac2a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:41:49.624 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:49.638+0000 7fac2a7fc640 10 monclient(hunting): my global_id is 4264 2024-03-29T10:41:49.624 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:49.638+0000 7fac2a7fc640 1 -- 172.21.15.88:0/517489519 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fac2c0cccb0 con 0x7fac2c029c30 2024-03-29T10:41:49.624 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:49.638+0000 7fac2a7fc640 1 -- 172.21.15.88:0/517489519 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1741403698 0 0) 0x7fac2c02ebf0 con 0x7fac2c02d890 2024-03-29T10:41:49.624 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:49.638+0000 7fac2a7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:41:49.625 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:49.638+0000 7fac2a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:41:49.625 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:49.638+0000 7fac2a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:41:49.625 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:49.638+0000 7fac2a7fc640 10 monclient(hunting): my global_id is 4278 2024-03-29T10:41:49.625 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:49.638+0000 7fac2a7fc640 1 -- 172.21.15.88:0/517489519 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fac2c02ee60 con 0x7fac2c02d890 2024-03-29T10:41:49.625 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:49.638+0000 7fac2a7fc640 1 -- 172.21.15.88:0/517489519 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1160807526 0 0) 0x7fac18004eb0 con 0x7fac2c006a70 2024-03-29T10:41:49.625 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:49.638+0000 7fac2a7fc640 1 -- 172.21.15.88:0/517489519 --> v1:172.21.15.88:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fac2c02ebf0 con 0x7fac2c006a70 2024-03-29T10:41:49.625 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:49.638+0000 7fac2a7fc640 1 -- 172.21.15.88:0/517489519 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7fac3404f400 con 0x7fac2c006a70 2024-03-29T10:41:49.625 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:49.639+0000 7fac2a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:49.625 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:49.639+0000 7fac2a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:41:49.625 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:49.639+0000 7fac2a7fc640 10 monclient(hunting): dump: 2024-03-29T10:41:49.625 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:41:49.625 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:41:49.625 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:41:49.625 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:41:49.626 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:49.626 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:41:49.626 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:41:49.626 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:41:49.626 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:41:49.626 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:41:49.626 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:49.639+0000 7fac2a7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:41:49.628 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:49.641+0000 7fac2a7fc640 1 -- 172.21.15.88:0/517489519 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 134394651 0 0) 0x7fac2c0cccb0 con 0x7fac2c029c30 2024-03-29T10:41:49.628 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:49.641+0000 7fac2a7fc640 1 -- 172.21.15.88:0/517489519 --> v1:172.21.15.196:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fac18004eb0 con 0x7fac2c029c30 2024-03-29T10:41:49.628 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:49.641+0000 7fac2a7fc640 1 -- 172.21.15.88:0/517489519 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7fac30001750 con 0x7fac2c029c30 2024-03-29T10:41:49.628 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:49.641+0000 7fac2a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:49.629 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:49.641+0000 7fac2a7fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T10:41:49.629 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:49.641+0000 7fac2a7fc640 10 monclient(hunting): dump: 2024-03-29T10:41:49.629 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:41:49.629 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:41:49.629 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:41:49.629 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:41:49.629 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:49.629 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:41:49.629 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:41:49.629 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:41:49.629 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:41:49.629 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:41:49.629 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:49.641+0000 7fac2a7fc640 1 -- 172.21.15.88:0/517489519 <== mon.2 v1:172.21.15.88:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 65802817 0 0) 0x7fac2c02ebf0 con 0x7fac2c006a70 2024-03-29T10:41:49.629 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:49.641+0000 7fac2a7fc640 1 -- 172.21.15.88:0/517489519 >> v1:172.21.15.196:6789/0 conn(0x7fac2c029c30 legacy=0x7fac2c169b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:49.630 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:49.641+0000 7fac2a7fc640 1 -- 172.21.15.88:0/517489519 >> v1:172.21.15.88:6789/0 conn(0x7fac2c02d890 legacy=0x7fac2c166760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:49.630 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:49.641+0000 7fac2a7fc640 10 monclient: _finish_hunting 0 2024-03-29T10:41:49.630 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:49.641+0000 7fac2a7fc640 1 monclient: found mon.c 2024-03-29T10:41:49.630 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:49.641+0000 7fac2a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:41:49.630 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:49.642+0000 7fac2a7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-03-29T10:41:49.630 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:49.642+0000 7fac2a7fc640 1 -- 172.21.15.88:0/517489519 --> v1:172.21.15.88:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac2c0cce40 con 0x7fac2c006a70 2024-03-29T10:41:49.630 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:49.642+0000 7fac2a7fc640 10 monclient: _finish_auth 0 2024-03-29T10:41:49.630 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:49.642+0000 7fac2a7fc640 10 monclient: _check_auth_tickets 2024-03-29T10:41:49.630 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:49.642+0000 7fac2a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:41:49.631 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:49.642+0000 7fac3a9f3640 5 monclient: authenticate success, global_id 4271 2024-03-29T10:41:49.631 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:49.642+0000 7fac3a9f3640 10 monclient: _renew_subs 2024-03-29T10:41:49.631 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:49.642+0000 7fac3a9f3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-03-29T10:41:49.631 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:49.642+0000 7fac3a9f3640 1 -- 172.21.15.88:0/517489519 --> v1:172.21.15.88:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac2c16a260 con 0x7fac2c006a70 2024-03-29T10:41:49.631 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:49.642+0000 7fac3a9f3640 10 client.4271.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:41:49.631 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:49.642+0000 7fac3a9f3640 10 monclient: _renew_subs 2024-03-29T10:41:49.631 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:49.642+0000 7fac3a9f3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-03-29T10:41:49.631 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:49.642+0000 7fac3a9f3640 1 -- 172.21.15.88:0/517489519 --> v1:172.21.15.88:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fac2c167050 con 0x7fac2c006a70 2024-03-29T10:41:49.631 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:49.642+0000 7fac3a9f3640 1 librados: init done 2024-03-29T10:41:49.631 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:49.644+0000 7fac3a9f3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:41:49.631 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:49.644+0000 7fac3a9f3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:41:49.631 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:49.644+0000 7fac3a9f3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-03-29T10:41:49.632 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:49.644+0000 7fac3a9f3640 1 -- 172.21.15.88:0/517489519 --> v1:172.21.15.88:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac2c16a450 con 0x7fac2c006a70 2024-03-29T10:41:49.632 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:49.645+0000 7fac2a7fc640 1 -- 172.21.15.88:0/517489519 <== mon.2 v1:172.21.15.88:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fac3404fdd0 con 0x7fac2c006a70 2024-03-29T10:41:49.632 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:49.645+0000 7fac2a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:41:49.632 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:49.645+0000 7fac2a7fc640 1 -- 172.21.15.88:0/517489519 <== mon.2 v1:172.21.15.88:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7fac34075710 con 0x7fac2c006a70 2024-03-29T10:41:49.632 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:49.645+0000 7fac2a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:49.632 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:49.645+0000 7fac2a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:41:49.632 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:49.645+0000 7fac2a7fc640 10 monclient: dump: 2024-03-29T10:41:49.632 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:41:49.632 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:41:49.632 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:41:49.632 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:41:49.632 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:49.632 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:41:49.632 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:41:49.633 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:41:49.633 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:41:49.633 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:41:49.633 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:49.647+0000 7fac2b7fe640 4 set_mon_vals no callback set 2024-03-29T10:41:49.635 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:49.649+0000 7fac2a7fc640 1 -- 172.21.15.88:0/517489519 <== mon.2 v1:172.21.15.88:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4235629220 0 0) 0x7fac34067ea0 con 0x7fac2c006a70 2024-03-29T10:41:49.635 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:49.649+0000 7fac2a7fc640 1 -- 172.21.15.88:0/517489519 <== mon.2 v1:172.21.15.88:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4257673582 0 0) 0x7fac340ff590 con 0x7fac2c006a70 2024-03-29T10:41:49.635 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:49.649+0000 7fac2a7fc640 10 client.4271.objecter ms_dispatch 0x7fac34000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:41:49.635 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:49.649+0000 7fac2a7fc640 3 client.4271.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:41:49.635 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:49.649+0000 7fac2a7fc640 3 client.4271.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:41:49.635 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:49.649+0000 7fac391f0640 10 monclient: get_auth_request con 0x7fac1806dfc0 auth_method 0 2024-03-29T10:41:49.635 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:49.649+0000 7fac2a7fc640 20 client.4271.objecter dump_active .. 0 homeless 2024-03-29T10:41:49.635 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:49.650+0000 7fac2a7fc640 10 client.4271.objecter ms_handle_connect 0x7fac1806dfc0 2024-03-29T10:41:49.642 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:49.657+0000 7fac2a7fc640 1 -- 172.21.15.88:0/517489519 <== mon.2 v1:172.21.15.88:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fac340cae70 con 0x7fac2c006a70 2024-03-29T10:41:49.642 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:49.657+0000 7fac2a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:41:49.642 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:49.657+0000 7fac2a7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:41:49.967 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:49.983+0000 7fac3a9f3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:41:49.967 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:49.984+0000 7fac3a9f3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:41:49.967 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:49.984+0000 7fac3a9f3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-03-29T10:41:49.967 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:49.984+0000 7fac3a9f3640 1 -- 172.21.15.88:0/517489519 --> v1:172.21.15.88:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fac2c004cd0 con 0x7fac2c006a70 2024-03-29T10:41:49.972 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:49.988+0000 7fac2a7fc640 1 -- 172.21.15.88:0/517489519 <== mon.2 v1:172.21.15.88:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 344506374) 0x7fac340ccda0 con 0x7fac2c006a70 2024-03-29T10:41:49.972 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:49.989+0000 7fac2a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:41:49.972 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:49.989+0000 7fac2a7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:41:49.974 INFO:teuthology.misc.health.smithi088.stdout: 2024-03-29T10:41:49.974 INFO:teuthology.misc.health.smithi088.stdout:{"epoch":10,"fsid":"869f5188-f92b-42dd-a46d-8c646e0388a9","created":"2024-03-29T10:40:40.285087+0000","modified":"2024-03-29T10:40:42.536230+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:40:42.536230+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a26c0c1c-8de0-438c-a3f6-3df10ea61574","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5de24915-8dec-49b6-8b16-f3170197c591","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ac054af3-d589-4565-9644-e6af75975c53","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3ec4f045-c714-4a22-bd1a-e07330b45ab3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"cddb9867-8e30-4951-9b13-3d1e78b9c45a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dc6d9911-785d-432f-8b53-56a1ccab302f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"77c88437-b587-496e-8b4d-1824ea8b5305","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ce963d96-5a8e-44c3-99de-c0dbebe234ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:41:49.976 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:49.993+0000 7fac3a9f3640 10 librados: watch_flush enter 2024-03-29T10:41:49.977 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:49.994+0000 7fac3a9f3640 10 librados: watch_flush exit 2024-03-29T10:41:49.977 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:49.994+0000 7fac3a9f3640 20 client.4271.objecter shutdown clearing up homeless session... 2024-03-29T10:41:49.977 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:49.994+0000 7fac3a9f3640 10 client.4271.objecter successfully canceled tick 2024-03-29T10:41:49.979 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:49.995+0000 7fac3a9f3640 1 -- 172.21.15.88:0/517489519 >> v1:172.21.15.88:6816/1752861349 conn(0x7fac1806dfc0 legacy=0x7fac18070480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:49.979 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:49.995+0000 7fac3a9f3640 10 monclient: shutdown 2024-03-29T10:41:49.979 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:49.995+0000 7fac3a9f3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:41:49.979 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:49.995+0000 7fac3a9f3640 1 -- 172.21.15.88:0/517489519 >> v1:172.21.15.88:6790/0 conn(0x7fac2c006a70 legacy=0x7fac2c02cb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:49.980 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:49.995+0000 7fac3a9f3640 1 -- 172.21.15.88:0/517489519 shutdown_connections 2024-03-29T10:41:49.980 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:49.995+0000 7fac3a9f3640 1 -- 172.21.15.88:0/517489519 >> 172.21.15.88:0/517489519 conn(0x7fac2c0058f0 msgr2=0x7fac2c0c3e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:41:49.980 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:49.995+0000 7fac3a9f3640 1 -- 172.21.15.88:0/517489519 shutdown_connections 2024-03-29T10:41:49.980 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:49.995+0000 7fac3a9f3640 1 -- 172.21.15.88:0/517489519 wait complete. 2024-03-29T10:41:49.980 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:49.996+0000 7fac3a9f3640 1 librados: shutdown 2024-03-29T10:41:49.998 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:41:56.805 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:41:56.965 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:56.981+0000 7f6ea9f7c640 10 monclient: get_monmap_and_config 2024-03-29T10:41:56.965 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:56.981+0000 7f6ea9f7c640 10 monclient: build_initial_monmap 2024-03-29T10:41:56.965 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:56.981+0000 7f6ea9f7c640 10 monclient: monmap: 2024-03-29T10:41:56.965 INFO:teuthology.misc.health.smithi088.stderr:epoch 0 2024-03-29T10:41:56.965 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:41:56.965 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:41:56.982396+0000 2024-03-29T10:41:56.965 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:41:56.982396+0000 2024-03-29T10:41:56.965 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 0 (unknown) 2024-03-29T10:41:56.965 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:41:56.965 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-03-29T10:41:56.966 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-29T10:41:56.966 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-b 2024-03-29T10:41:56.966 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:41:56.968 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:56.982+0000 7f6ea9f7c640 1 Processor -- start 2024-03-29T10:41:56.968 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:56.982+0000 7f6ea9f7c640 1 -- start start 2024-03-29T10:41:56.968 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:56.982+0000 7f6ea9f7c640 10 monclient: init 2024-03-29T10:41:56.968 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:56.983+0000 7f6ea9f7c640 10 monclient: _reopen_session rank -1 2024-03-29T10:41:56.968 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:56.983+0000 7f6ea9f7c640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:41:56.968 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:56.983+0000 7f6ea9f7c640 10 monclient(hunting): picked mon.noname-a con 0x7f6ea41422b0 addr v1:172.21.15.88:6789/0 2024-03-29T10:41:56.968 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:56.983+0000 7f6ea9f7c640 10 monclient(hunting): picked mon.noname-c con 0x7f6ea4142690 addr v1:172.21.15.196:6789/0 2024-03-29T10:41:56.968 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:56.983+0000 7f6ea9f7c640 10 monclient(hunting): picked mon.noname-b con 0x7f6ea40b17d0 addr v1:172.21.15.88:6790/0 2024-03-29T10:41:56.968 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:56.983+0000 7f6ea9f7c640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ea4142c80 con 0x7f6ea41422b0 2024-03-29T10:41:56.968 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:56.983+0000 7f6ea9f7c640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ea40aae40 con 0x7f6ea4142690 2024-03-29T10:41:56.968 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:56.983+0000 7f6ea9f7c640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ea413fbd0 con 0x7f6ea40b17d0 2024-03-29T10:41:56.968 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:56.983+0000 7f6ea9f7c640 10 monclient(hunting): _renew_subs 2024-03-29T10:41:56.969 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:56.983+0000 7f6ea9f7c640 10 monclient(hunting): authenticate will time out at 839.219651s 2024-03-29T10:41:56.969 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:56.984+0000 7f6e9bfff640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f6ea4142690 0x7f6ea40b2590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.88:47830/0 (socket says 172.21.15.88:47830) 2024-03-29T10:41:56.969 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:56.984+0000 7f6e9bfff640 1 -- 172.21.15.88:0/3427610823 learned_addr learned my addr 172.21.15.88:0/3427610823 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-29T10:41:56.969 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:56.984+0000 7f6ea2ffd640 1 -- 172.21.15.88:0/3427610823 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1760584819 0 0) 0x7f6ea40aae40 con 0x7f6ea4142690 2024-03-29T10:41:56.969 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:56.984+0000 7f6ea2ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:41:56.969 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:56.984+0000 7f6ea2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:41:56.969 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:56.984+0000 7f6ea2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:41:56.969 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:56.984+0000 7f6ea2ffd640 10 monclient(hunting): my global_id is 4267 2024-03-29T10:41:56.969 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:56.984+0000 7f6ea2ffd640 1 -- 172.21.15.88:0/3427610823 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e90003520 con 0x7f6ea4142690 2024-03-29T10:41:56.969 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:56.984+0000 7f6ea2ffd640 1 -- 172.21.15.88:0/3427610823 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1910406079 0 0) 0x7f6ea413fbd0 con 0x7f6ea40b17d0 2024-03-29T10:41:56.969 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:56.984+0000 7f6ea2ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:41:56.969 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:56.984+0000 7f6ea2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:41:56.969 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:56.984+0000 7f6ea2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:41:56.969 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:56.984+0000 7f6ea2ffd640 10 monclient(hunting): my global_id is 4274 2024-03-29T10:41:56.970 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:56.984+0000 7f6ea2ffd640 1 -- 172.21.15.88:0/3427610823 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e90003840 con 0x7f6ea40b17d0 2024-03-29T10:41:56.970 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:56.985+0000 7f6ea2ffd640 1 -- 172.21.15.88:0/3427610823 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2176918064 0 0) 0x7f6e90003520 con 0x7f6ea4142690 2024-03-29T10:41:56.970 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:56.985+0000 7f6ea2ffd640 1 -- 172.21.15.88:0/3427610823 >> v1:172.21.15.88:6790/0 conn(0x7f6ea40b17d0 legacy=0x7f6ea40b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:56.971 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:56.985+0000 7f6ea2ffd640 1 -- 172.21.15.88:0/3427610823 >> v1:172.21.15.88:6789/0 conn(0x7f6ea41422b0 legacy=0x7f6ea413f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:56.971 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:56.985+0000 7f6ea2ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:41:56.971 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:56.985+0000 7f6ea2ffd640 1 monclient: found mon.noname-c 2024-03-29T10:41:56.971 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:56.985+0000 7f6ea2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:41:56.971 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:56.985+0000 7f6ea2ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.196:6789/0 2024-03-29T10:41:56.971 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:56.985+0000 7f6ea2ffd640 1 -- 172.21.15.88:0/3427610823 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ea4151940 con 0x7f6ea4142690 2024-03-29T10:41:56.971 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:56.985+0000 7f6ea2ffd640 10 monclient: _finish_auth 0 2024-03-29T10:41:56.971 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:56.985+0000 7f6ea2ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:41:56.971 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:56.985+0000 7f6ea2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:41:56.971 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:56.985+0000 7f6ea2ffd640 1 -- 172.21.15.88:0/3427610823 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f6e8c001670 con 0x7f6ea4142690 2024-03-29T10:41:56.971 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:56.985+0000 7f6ea2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:56.971 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:56.985+0000 7f6ea2ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:41:56.971 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:56.985+0000 7f6ea2ffd640 10 monclient: dump: 2024-03-29T10:41:56.972 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:41:56.972 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:41:56.972 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:41:56.972 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:41:56.972 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:56.972 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:41:56.972 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:41:56.972 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:41:56.972 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:41:56.972 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:41:56.972 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:56.985+0000 7f6ea9f7c640 5 monclient: authenticate success, global_id 4267 2024-03-29T10:41:56.972 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:56.985+0000 7f6ea9f7c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:41:56.972 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:56.986+0000 7f6ea2ffd640 1 -- 172.21.15.88:0/3427610823 <== mon.1 v1:172.21.15.196:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6e8c0017d0 con 0x7f6ea4142690 2024-03-29T10:41:56.973 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:56.986+0000 7f6ea2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:41:56.973 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:56.986+0000 7f6ea2ffd640 1 -- 172.21.15.88:0/3427610823 <== mon.1 v1:172.21.15.196:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f6e8c001b60 con 0x7f6ea4142690 2024-03-29T10:41:56.973 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:56.986+0000 7f6ea2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:56.973 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:56.986+0000 7f6ea2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:41:56.973 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:56.986+0000 7f6ea2ffd640 10 monclient: dump: 2024-03-29T10:41:56.973 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:41:56.973 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:41:56.973 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:41:56.973 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:41:56.973 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:56.973 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:41:56.973 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:41:56.973 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:41:56.974 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:41:56.974 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:41:56.974 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:56.986+0000 7f6ea9f7c640 10 monclient: get_monmap_and_config success 2024-03-29T10:41:56.974 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:56.986+0000 7f6ea9f7c640 4 set_mon_vals no callback set 2024-03-29T10:41:56.974 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:56.987+0000 7f6ea9f7c640 10 monclient: shutdown 2024-03-29T10:41:56.974 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:56.987+0000 7f6ea9f7c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:41:56.974 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:56.987+0000 7f6ea9f7c640 1 -- 172.21.15.88:0/3427610823 >> v1:172.21.15.196:6789/0 conn(0x7f6ea4142690 legacy=0x7f6ea40b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:56.980 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:56.987+0000 7f6ea9f7c640 1 -- 172.21.15.88:0/3427610823 shutdown_connections 2024-03-29T10:41:56.981 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:56.987+0000 7f6ea9f7c640 1 -- 172.21.15.88:0/3427610823 >> 172.21.15.88:0/3427610823 conn(0x7f6ea4066a00 msgr2=0x7f6ea4066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:41:56.981 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:56.988+0000 7f6ea9f7c640 1 -- 172.21.15.88:0/3427610823 shutdown_connections 2024-03-29T10:41:56.981 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:56.988+0000 7f6ea9f7c640 1 -- 172.21.15.88:0/3427610823 wait complete. 2024-03-29T10:41:56.981 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:56.989+0000 7f6ea9f7c640 10 monclient: build_initial_monmap 2024-03-29T10:41:56.981 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:56.989+0000 7f6ea9f7c640 10 monclient: monmap: 2024-03-29T10:41:56.981 INFO:teuthology.misc.health.smithi088.stderr:epoch 0 2024-03-29T10:41:56.981 INFO:teuthology.misc.health.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:41:56.981 INFO:teuthology.misc.health.smithi088.stderr:last_changed 0.000000 2024-03-29T10:41:56.981 INFO:teuthology.misc.health.smithi088.stderr:created 0.000000 2024-03-29T10:41:56.981 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 0 (unknown) 2024-03-29T10:41:56.981 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:41:56.981 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-03-29T10:41:56.981 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-29T10:41:56.982 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-c 2024-03-29T10:41:56.982 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:41:56.982 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:56.989+0000 7f6ea9f7c640 1 librados: starting msgr at 2024-03-29T10:41:56.982 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:56.989+0000 7f6ea9f7c640 1 librados: starting objecter 2024-03-29T10:41:56.982 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:56.989+0000 7f6ea9f7c640 1 Processor -- start 2024-03-29T10:41:56.982 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:56.989+0000 7f6ea9f7c640 1 -- start start 2024-03-29T10:41:56.982 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:56.989+0000 7f6ea9f7c640 1 librados: setting wanted keys 2024-03-29T10:41:56.982 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:56.989+0000 7f6ea9f7c640 1 librados: calling monclient init 2024-03-29T10:41:56.982 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:56.989+0000 7f6ea9f7c640 10 monclient: init 2024-03-29T10:41:56.982 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:56.990+0000 7f6ea9f7c640 10 monclient: _reopen_session rank -1 2024-03-29T10:41:56.982 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:56.991+0000 7f6ea9f7c640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:41:56.982 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:56.991+0000 7f6ea9f7c640 10 monclient(hunting): picked mon.noname-c con 0x7f6ea40b17d0 addr v1:172.21.15.88:6790/0 2024-03-29T10:41:56.983 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:56.991+0000 7f6ea37fe640 1 --1- >> v1:172.21.15.88:6790/0 conn(0x7f6ea40b17d0 0x7f6ea41dab10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6790/0 says I am v1:172.21.15.88:33968/0 (socket says 172.21.15.88:33968) 2024-03-29T10:41:56.983 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:56.991+0000 7f6ea9f7c640 10 monclient(hunting): picked mon.noname-b con 0x7f6ea41422b0 addr v1:172.21.15.196:6789/0 2024-03-29T10:41:56.983 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:56.991+0000 7f6ea37fe640 1 -- 172.21.15.88:0/3180694251 learned_addr learned my addr 172.21.15.88:0/3180694251 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-29T10:41:56.983 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:56.991+0000 7f6ea9f7c640 10 monclient(hunting): picked mon.noname-a con 0x7f6ea4142690 addr v1:172.21.15.88:6789/0 2024-03-29T10:41:56.983 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:56.991+0000 7f6ea9f7c640 1 -- 172.21.15.88:0/3180694251 --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ea40b7310 con 0x7f6ea4142690 2024-03-29T10:41:56.983 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:56.991+0000 7f6ea9f7c640 1 -- 172.21.15.88:0/3180694251 --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ea412e670 con 0x7f6ea41422b0 2024-03-29T10:41:56.983 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:56.991+0000 7f6ea9f7c640 1 -- 172.21.15.88:0/3180694251 --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ea41db220 con 0x7f6ea40b17d0 2024-03-29T10:41:56.983 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:56.991+0000 7f6ea9f7c640 10 monclient(hunting): _renew_subs 2024-03-29T10:41:56.983 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:56.991+0000 7f6ea9f7c640 10 monclient(hunting): authenticate will time out at 839.227301s 2024-03-29T10:41:56.983 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:56.991+0000 7f6ea0ff9640 10 client.?.objecter ms_handle_connect 0x7f6ea40b17d0 2024-03-29T10:41:56.983 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:56.991+0000 7f6ea0ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:41:56.983 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:56.991+0000 7f6ea0ff9640 10 client.?.objecter ms_handle_connect 0x7f6ea41422b0 2024-03-29T10:41:56.983 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:56.991+0000 7f6ea0ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:41:56.984 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:56.992+0000 7f6ea0ff9640 1 -- 172.21.15.88:0/3180694251 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2699702758 0 0) 0x7f6ea412e670 con 0x7f6ea41422b0 2024-03-29T10:41:56.984 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:56.992+0000 7f6ea0ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:41:56.984 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:56.992+0000 7f6ea0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:41:56.984 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:56.992+0000 7f6ea0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:41:56.984 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:56.992+0000 7f6ea0ff9640 10 monclient(hunting): my global_id is 4270 2024-03-29T10:41:56.984 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:56.992+0000 7f6ea0ff9640 1 -- 172.21.15.88:0/3180694251 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e84004eb0 con 0x7f6ea41422b0 2024-03-29T10:41:56.984 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:56.992+0000 7f6ea0ff9640 10 client.?.objecter ms_handle_connect 0x7f6ea4142690 2024-03-29T10:41:56.984 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:56.992+0000 7f6ea0ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:41:56.984 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:56.992+0000 7f6ea0ff9640 1 -- 172.21.15.88:0/3180694251 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 331866576 0 0) 0x7f6e84004eb0 con 0x7f6ea41422b0 2024-03-29T10:41:56.984 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:56.992+0000 7f6ea0ff9640 1 -- 172.21.15.88:0/3180694251 --> v1:172.21.15.196:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6ea412e670 con 0x7f6ea41422b0 2024-03-29T10:41:56.984 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:56.993+0000 7f6ea0ff9640 1 -- 172.21.15.88:0/3180694251 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 460480394 0 0) 0x7f6ea41db220 con 0x7f6ea40b17d0 2024-03-29T10:41:56.984 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:56.993+0000 7f6ea0ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:41:56.984 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:56.993+0000 7f6ea0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:41:56.984 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:56.993+0000 7f6ea0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:41:56.985 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:56.993+0000 7f6ea0ff9640 10 monclient(hunting): my global_id is 4277 2024-03-29T10:41:56.985 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:56.993+0000 7f6ea0ff9640 1 -- 172.21.15.88:0/3180694251 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e84004eb0 con 0x7f6ea40b17d0 2024-03-29T10:41:56.985 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:56.993+0000 7f6ea0ff9640 1 -- 172.21.15.88:0/3180694251 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f6e8c001ae0 con 0x7f6ea41422b0 2024-03-29T10:41:56.985 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:56.993+0000 7f6ea0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:56.985 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:56.993+0000 7f6ea0ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:41:56.985 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:56.993+0000 7f6ea0ff9640 10 monclient(hunting): dump: 2024-03-29T10:41:56.985 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:41:56.985 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:41:56.985 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:41:56.985 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:41:56.985 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:56.985 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:41:56.985 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:41:56.985 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:41:56.986 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:41:56.986 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:41:56.986 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:56.993+0000 7f6ea0ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:41:56.986 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:56.993+0000 7f6ea0ff9640 1 -- 172.21.15.88:0/3180694251 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4202061776 0 0) 0x7f6ea40b7310 con 0x7f6ea4142690 2024-03-29T10:41:56.986 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:56.993+0000 7f6ea0ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:41:56.986 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:56.993+0000 7f6ea0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:41:56.986 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:56.993+0000 7f6ea0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:41:56.986 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:56.993+0000 7f6ea0ff9640 10 monclient(hunting): my global_id is 4284 2024-03-29T10:41:56.986 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:56.993+0000 7f6ea0ff9640 1 -- 172.21.15.88:0/3180694251 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6ea41db220 con 0x7f6ea4142690 2024-03-29T10:41:56.986 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:56.994+0000 7f6ea0ff9640 1 -- 172.21.15.88:0/3180694251 <== mon.1 v1:172.21.15.196:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2779036886 0 0) 0x7f6ea412e670 con 0x7f6ea41422b0 2024-03-29T10:41:56.986 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:56.994+0000 7f6ea0ff9640 1 -- 172.21.15.88:0/3180694251 >> v1:172.21.15.88:6790/0 conn(0x7f6ea40b17d0 legacy=0x7f6ea41dab10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:56.986 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:56.994+0000 7f6ea0ff9640 1 -- 172.21.15.88:0/3180694251 >> v1:172.21.15.88:6789/0 conn(0x7f6ea4142690 legacy=0x7f6ea412df60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:56.986 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:56.994+0000 7f6ea0ff9640 10 monclient: _finish_hunting 0 2024-03-29T10:41:56.987 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:56.994+0000 7f6ea0ff9640 1 monclient: found mon.b 2024-03-29T10:41:56.987 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:56.994+0000 7f6ea0ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:41:56.987 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:56.994+0000 7f6ea0ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T10:41:56.987 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:56.994+0000 7f6ea0ff9640 1 -- 172.21.15.88:0/3180694251 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ea40b5650 con 0x7f6ea41422b0 2024-03-29T10:41:56.987 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:56.994+0000 7f6ea0ff9640 10 monclient: _finish_auth 0 2024-03-29T10:41:56.987 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:56.994+0000 7f6ea0ff9640 10 monclient: _check_auth_tickets 2024-03-29T10:41:56.987 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:56.994+0000 7f6ea0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:41:56.987 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:56.994+0000 7f6ea9f7c640 5 monclient: authenticate success, global_id 4270 2024-03-29T10:41:56.987 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:56.994+0000 7f6ea9f7c640 10 monclient: _renew_subs 2024-03-29T10:41:56.987 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:56.994+0000 7f6ea9f7c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T10:41:56.987 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:56.994+0000 7f6ea9f7c640 1 -- 172.21.15.88:0/3180694251 --> v1:172.21.15.196:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ea4151060 con 0x7f6ea41422b0 2024-03-29T10:41:56.987 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:56.994+0000 7f6ea9f7c640 10 client.4270.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:41:56.987 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:56.994+0000 7f6ea9f7c640 10 monclient: _renew_subs 2024-03-29T10:41:56.987 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:56.994+0000 7f6ea9f7c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T10:41:56.988 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:56.994+0000 7f6ea9f7c640 1 -- 172.21.15.88:0/3180694251 --> v1:172.21.15.196:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ea40b78a0 con 0x7f6ea41422b0 2024-03-29T10:41:56.988 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:56.994+0000 7f6ea9f7c640 1 librados: init done 2024-03-29T10:41:56.988 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:56.995+0000 7f6ea0ff9640 1 -- 172.21.15.88:0/3180694251 <== mon.1 v1:172.21.15.196:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6e8c004a90 con 0x7f6ea41422b0 2024-03-29T10:41:56.988 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:56.995+0000 7f6ea0ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:41:56.988 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:56.995+0000 7f6ea0ff9640 1 -- 172.21.15.88:0/3180694251 <== mon.1 v1:172.21.15.196:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f6e8c004da0 con 0x7f6ea41422b0 2024-03-29T10:41:56.988 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:56.995+0000 7f6ea0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:56.988 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:56.995+0000 7f6ea0ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:41:56.988 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:56.995+0000 7f6ea0ff9640 10 monclient: dump: 2024-03-29T10:41:56.988 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:41:56.988 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:41:56.988 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:41:56.988 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:41:56.988 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:56.988 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:41:56.989 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:41:56.989 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:41:56.989 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:41:56.989 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:41:56.989 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:56.995+0000 7f6ea27fc640 4 set_mon_vals no callback set 2024-03-29T10:41:56.989 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:56.995+0000 7f6ea9f7c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:41:56.989 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:56.996+0000 7f6ea9f7c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:41:56.989 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:56.996+0000 7f6ea9f7c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T10:41:56.989 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:56.996+0000 7f6ea9f7c640 1 -- 172.21.15.88:0/3180694251 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e88005080 con 0x7f6ea41422b0 2024-03-29T10:41:56.989 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:56.997+0000 7f6ea0ff9640 1 -- 172.21.15.88:0/3180694251 <== mon.1 v1:172.21.15.196:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4235629220 0 0) 0x7f6e8c01ad70 con 0x7f6ea41422b0 2024-03-29T10:41:56.990 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:56.998+0000 7f6ea37fe640 10 monclient: get_auth_request con 0x7f6e8406def0 auth_method 0 2024-03-29T10:41:56.990 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:56.998+0000 7f6ea0ff9640 1 -- 172.21.15.88:0/3180694251 <== mon.1 v1:172.21.15.196:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4257673582 0 0) 0x7f6e8c084a60 con 0x7f6ea41422b0 2024-03-29T10:41:56.990 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:56.998+0000 7f6ea0ff9640 10 client.4270.objecter ms_dispatch 0x7f6ea4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:41:56.990 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:56.998+0000 7f6ea0ff9640 3 client.4270.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:41:56.990 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:56.998+0000 7f6ea0ff9640 3 client.4270.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:41:56.990 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:56.998+0000 7f6ea0ff9640 20 client.4270.objecter dump_active .. 0 homeless 2024-03-29T10:41:56.990 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:56.998+0000 7f6ea0ff9640 10 client.4270.objecter ms_handle_connect 0x7f6e8406def0 2024-03-29T10:41:56.990 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:57.004+0000 7f6ea0ff9640 1 -- 172.21.15.88:0/3180694251 <== mon.1 v1:172.21.15.196:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6e8c050340 con 0x7f6ea41422b0 2024-03-29T10:41:56.990 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:57.005+0000 7f6ea0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:41:56.990 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:57.005+0000 7f6ea0ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:41:57.379 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:57.396+0000 7f6ea9f7c640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:41:57.379 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:57.396+0000 7f6ea9f7c640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:41:57.380 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:57.396+0000 7f6ea9f7c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T10:41:57.380 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:57.396+0000 7f6ea9f7c640 1 -- 172.21.15.88:0/3180694251 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6e88005330 con 0x7f6ea41422b0 2024-03-29T10:41:57.380 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:57.397+0000 7f6ea0ff9640 1 -- 172.21.15.88:0/3180694251 <== mon.1 v1:172.21.15.196:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 344506374) 0x7f6e8c052270 con 0x7f6ea41422b0 2024-03-29T10:41:57.380 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:57.397+0000 7f6ea0ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:41:57.381 INFO:teuthology.misc.health.smithi088.stdout: 2024-03-29T10:41:57.381 INFO:teuthology.misc.health.smithi088.stdout:{"epoch":10,"fsid":"869f5188-f92b-42dd-a46d-8c646e0388a9","created":"2024-03-29T10:40:40.285087+0000","modified":"2024-03-29T10:40:42.536230+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:40:42.536230+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a26c0c1c-8de0-438c-a3f6-3df10ea61574","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5de24915-8dec-49b6-8b16-f3170197c591","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ac054af3-d589-4565-9644-e6af75975c53","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3ec4f045-c714-4a22-bd1a-e07330b45ab3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"cddb9867-8e30-4951-9b13-3d1e78b9c45a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dc6d9911-785d-432f-8b53-56a1ccab302f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"77c88437-b587-496e-8b4d-1824ea8b5305","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ce963d96-5a8e-44c3-99de-c0dbebe234ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:41:57.381 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:57.397+0000 7f6ea0ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:41:57.385 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:57.400+0000 7f6ea9f7c640 10 librados: watch_flush enter 2024-03-29T10:41:57.385 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:57.400+0000 7f6ea9f7c640 10 librados: watch_flush exit 2024-03-29T10:41:57.385 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:57.400+0000 7f6ea9f7c640 20 client.4270.objecter shutdown clearing up homeless session... 2024-03-29T10:41:57.385 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:57.401+0000 7f6ea9f7c640 10 client.4270.objecter successfully canceled tick 2024-03-29T10:41:57.385 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:57.401+0000 7f6ea9f7c640 1 -- 172.21.15.88:0/3180694251 >> v1:172.21.15.88:6816/1752861349 conn(0x7f6e8406def0 legacy=0x7f6e840703b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:57.385 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:57.401+0000 7f6ea9f7c640 10 monclient: shutdown 2024-03-29T10:41:57.385 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:57.401+0000 7f6ea9f7c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:41:57.385 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:57.401+0000 7f6ea9f7c640 1 -- 172.21.15.88:0/3180694251 >> v1:172.21.15.196:6789/0 conn(0x7f6ea41422b0 legacy=0x7f6ea412d850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:57.385 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:57.401+0000 7f6ea9f7c640 1 -- 172.21.15.88:0/3180694251 shutdown_connections 2024-03-29T10:41:57.385 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:57.401+0000 7f6ea9f7c640 1 -- 172.21.15.88:0/3180694251 >> 172.21.15.88:0/3180694251 conn(0x7f6ea4066a00 msgr2=0x7f6ea40b2490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:41:57.385 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:57.401+0000 7f6ea9f7c640 1 -- 172.21.15.88:0/3180694251 shutdown_connections 2024-03-29T10:41:57.385 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:57.401+0000 7f6ea9f7c640 1 -- 172.21.15.88:0/3180694251 wait complete. 2024-03-29T10:41:57.386 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:41:57.401+0000 7f6ea9f7c640 1 librados: shutdown 2024-03-29T10:41:57.400 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:42:04.207 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:42:04.331 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:04.348+0000 7f40c8845640 10 monclient: get_monmap_and_config 2024-03-29T10:42:04.332 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:04.348+0000 7f40c8845640 10 monclient: build_initial_monmap 2024-03-29T10:42:04.332 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:04.348+0000 7f40c8845640 10 monclient: monmap: 2024-03-29T10:42:04.332 INFO:teuthology.misc.health.smithi088.stderr:epoch 0 2024-03-29T10:42:04.332 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:42:04.332 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:42:04.349409+0000 2024-03-29T10:42:04.332 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:42:04.349409+0000 2024-03-29T10:42:04.332 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 0 (unknown) 2024-03-29T10:42:04.332 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:42:04.332 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-03-29T10:42:04.332 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-29T10:42:04.332 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-b 2024-03-29T10:42:04.332 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:42:04.335 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:04.348+0000 7f40c8845640 1 Processor -- start 2024-03-29T10:42:04.335 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:04.349+0000 7f40c8845640 1 -- start start 2024-03-29T10:42:04.335 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:04.349+0000 7f40c8845640 10 monclient: init 2024-03-29T10:42:04.335 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:04.349+0000 7f40c8845640 10 monclient: _reopen_session rank -1 2024-03-29T10:42:04.335 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:04.349+0000 7f40c8845640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:42:04.335 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:04.349+0000 7f40c8845640 10 monclient(hunting): picked mon.noname-a con 0x7f40c41457f0 addr v1:172.21.15.88:6789/0 2024-03-29T10:42:04.336 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:04.350+0000 7f40c8845640 10 monclient(hunting): picked mon.noname-c con 0x7f40c413f650 addr v1:172.21.15.196:6789/0 2024-03-29T10:42:04.336 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:04.350+0000 7f40c1d9b640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f40c413f650 0x7f40c40b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.88:35236/0 (socket says 172.21.15.88:35236) 2024-03-29T10:42:04.336 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:04.350+0000 7f40c2d9d640 1 --1- >> v1:172.21.15.88:6790/0 conn(0x7f40c40a9d70 0x7f40c414d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6790/0 says I am v1:172.21.15.88:45012/0 (socket says 172.21.15.88:45012) 2024-03-29T10:42:04.336 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:04.350+0000 7f40c1d9b640 1 -- 172.21.15.88:0/2102855659 learned_addr learned my addr 172.21.15.88:0/2102855659 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-29T10:42:04.336 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:04.350+0000 7f40c8845640 10 monclient(hunting): picked mon.noname-b con 0x7f40c40a9d70 addr v1:172.21.15.88:6790/0 2024-03-29T10:42:04.336 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:04.350+0000 7f40c8845640 1 -- 172.21.15.88:0/2102855659 --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40c40aaef0 con 0x7f40c41457f0 2024-03-29T10:42:04.336 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:04.350+0000 7f40c8845640 1 -- 172.21.15.88:0/2102855659 --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40c4145bd0 con 0x7f40c413f650 2024-03-29T10:42:04.336 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:04.351+0000 7f40c8845640 1 -- 172.21.15.88:0/2102855659 --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40c4145eb0 con 0x7f40c40a9d70 2024-03-29T10:42:04.336 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:04.351+0000 7f40c8845640 10 monclient(hunting): _renew_subs 2024-03-29T10:42:04.336 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:04.351+0000 7f40c8845640 10 monclient(hunting): authenticate will time out at 846.587111s 2024-03-29T10:42:04.336 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:04.351+0000 7f40c159a640 1 -- 172.21.15.88:0/2102855659 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4027342818 0 0) 0x7f40c4145eb0 con 0x7f40c40a9d70 2024-03-29T10:42:04.336 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:04.351+0000 7f40c159a640 10 monclient(hunting): _negotiate 2024-03-29T10:42:04.336 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:04.351+0000 7f40c159a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:42:04.337 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:04.351+0000 7f40c159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:42:04.337 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:04.351+0000 7f40c159a640 10 monclient(hunting): my global_id is 4280 2024-03-29T10:42:04.337 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:04.351+0000 7f40c159a640 1 -- 172.21.15.88:0/2102855659 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f40ac003520 con 0x7f40c40a9d70 2024-03-29T10:42:04.337 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:04.351+0000 7f40c159a640 1 -- 172.21.15.88:0/2102855659 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 85852409 0 0) 0x7f40ac003520 con 0x7f40c40a9d70 2024-03-29T10:42:04.337 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:04.351+0000 7f40c159a640 1 -- 172.21.15.88:0/2102855659 >> v1:172.21.15.196:6789/0 conn(0x7f40c413f650 legacy=0x7f40c40b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:04.337 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:04.351+0000 7f40c159a640 1 -- 172.21.15.88:0/2102855659 >> v1:172.21.15.88:6789/0 conn(0x7f40c41457f0 legacy=0x7f40c413ef40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:42:04.339 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:04.353+0000 7f40c159a640 10 monclient: _finish_hunting 0 2024-03-29T10:42:04.339 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:04.353+0000 7f40c159a640 1 monclient: found mon.noname-b 2024-03-29T10:42:04.339 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:04.353+0000 7f40c159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:42:04.339 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:04.353+0000 7f40c159a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.88:6790/0 2024-03-29T10:42:04.339 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:04.353+0000 7f40c159a640 1 -- 172.21.15.88:0/2102855659 --> v1:172.21.15.88:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40c4146340 con 0x7f40c40a9d70 2024-03-29T10:42:04.339 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:04.353+0000 7f40c159a640 10 monclient: _finish_auth 0 2024-03-29T10:42:04.339 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:04.353+0000 7f40c159a640 10 monclient: _check_auth_tickets 2024-03-29T10:42:04.339 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:04.353+0000 7f40c159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:42:04.339 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:04.353+0000 7f40c159a640 1 -- 172.21.15.88:0/2102855659 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f40bc001940 con 0x7f40c40a9d70 2024-03-29T10:42:04.339 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:04.353+0000 7f40c159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:04.339 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:04.353+0000 7f40c159a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:42:04.339 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:04.353+0000 7f40c159a640 10 monclient: dump: 2024-03-29T10:42:04.340 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:42:04.340 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:42:04.340 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:42:04.340 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:42:04.340 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:04.340 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:42:04.340 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:42:04.340 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:42:04.340 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:42:04.340 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:42:04.340 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:04.353+0000 7f40c8845640 5 monclient: authenticate success, global_id 4280 2024-03-29T10:42:04.340 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:04.353+0000 7f40c8845640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:42:04.340 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:04.353+0000 7f40c159a640 1 -- 172.21.15.88:0/2102855659 <== mon.2 v1:172.21.15.88:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f40bc001b10 con 0x7f40c40a9d70 2024-03-29T10:42:04.341 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:04.353+0000 7f40c159a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:42:04.341 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:04.353+0000 7f40c159a640 1 -- 172.21.15.88:0/2102855659 <== mon.2 v1:172.21.15.88:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f40bc001f60 con 0x7f40c40a9d70 2024-03-29T10:42:04.341 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:04.353+0000 7f40c159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:04.341 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:04.353+0000 7f40c159a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:42:04.341 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:04.353+0000 7f40c159a640 10 monclient: dump: 2024-03-29T10:42:04.341 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:42:04.341 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:42:04.341 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:42:04.341 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:42:04.341 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:04.341 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:42:04.341 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:42:04.341 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:42:04.341 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:42:04.342 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:42:04.342 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:04.353+0000 7f40c8845640 10 monclient: get_monmap_and_config success 2024-03-29T10:42:04.342 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:04.353+0000 7f40c8845640 4 set_mon_vals no callback set 2024-03-29T10:42:04.342 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:04.354+0000 7f40c8845640 10 monclient: shutdown 2024-03-29T10:42:04.342 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:04.354+0000 7f40c8845640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:42:04.342 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:04.354+0000 7f40c8845640 1 -- 172.21.15.88:0/2102855659 >> v1:172.21.15.88:6790/0 conn(0x7f40c40a9d70 legacy=0x7f40c414d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:04.342 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:04.354+0000 7f40c8845640 1 -- 172.21.15.88:0/2102855659 shutdown_connections 2024-03-29T10:42:04.342 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:04.354+0000 7f40c8845640 1 -- 172.21.15.88:0/2102855659 >> 172.21.15.88:0/2102855659 conn(0x7f40c4066a00 msgr2=0x7f40c4066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:42:04.342 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:04.354+0000 7f40c8845640 1 -- 172.21.15.88:0/2102855659 shutdown_connections 2024-03-29T10:42:04.342 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:04.354+0000 7f40c8845640 1 -- 172.21.15.88:0/2102855659 wait complete. 2024-03-29T10:42:04.343 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:04.355+0000 7f40c8845640 10 monclient: build_initial_monmap 2024-03-29T10:42:04.343 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:04.355+0000 7f40c8845640 10 monclient: monmap: 2024-03-29T10:42:04.343 INFO:teuthology.misc.health.smithi088.stderr:epoch 0 2024-03-29T10:42:04.343 INFO:teuthology.misc.health.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:42:04.343 INFO:teuthology.misc.health.smithi088.stderr:last_changed 0.000000 2024-03-29T10:42:04.343 INFO:teuthology.misc.health.smithi088.stderr:created 0.000000 2024-03-29T10:42:04.343 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 0 (unknown) 2024-03-29T10:42:04.343 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:42:04.343 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-03-29T10:42:04.343 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-29T10:42:04.343 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-c 2024-03-29T10:42:04.343 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:42:04.348 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:04.356+0000 7f40c8845640 1 librados: starting msgr at 2024-03-29T10:42:04.348 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:04.356+0000 7f40c8845640 1 librados: starting objecter 2024-03-29T10:42:04.348 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:04.356+0000 7f40c8845640 1 Processor -- start 2024-03-29T10:42:04.348 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:04.356+0000 7f40c8845640 1 -- start start 2024-03-29T10:42:04.348 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:04.356+0000 7f40c8845640 1 librados: setting wanted keys 2024-03-29T10:42:04.348 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:04.356+0000 7f40c8845640 1 librados: calling monclient init 2024-03-29T10:42:04.348 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:04.356+0000 7f40c8845640 10 monclient: init 2024-03-29T10:42:04.348 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:04.357+0000 7f40c8845640 10 monclient: _reopen_session rank -1 2024-03-29T10:42:04.349 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:04.357+0000 7f40c8845640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:42:04.349 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:04.357+0000 7f40c8845640 10 monclient(hunting): picked mon.noname-c con 0x7f40c40a9d70 addr v1:172.21.15.88:6790/0 2024-03-29T10:42:04.349 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:04.357+0000 7f40c8845640 10 monclient(hunting): picked mon.noname-b con 0x7f40c413f650 addr v1:172.21.15.196:6789/0 2024-03-29T10:42:04.349 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:04.357+0000 7f40c259c640 1 --1- >> v1:172.21.15.88:6790/0 conn(0x7f40c40a9d70 0x7f40c413fda0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6790/0 says I am v1:172.21.15.88:45850/0 (socket says 172.21.15.88:45850) 2024-03-29T10:42:04.349 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:04.358+0000 7f40c1d9b640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f40c413f650 0x7f40c41419b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.88:36516/0 (socket says 172.21.15.88:36516) 2024-03-29T10:42:04.349 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:04.358+0000 7f40c8845640 10 monclient(hunting): picked mon.noname-a con 0x7f40c41457f0 addr v1:172.21.15.88:6789/0 2024-03-29T10:42:04.349 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:04.358+0000 7f40c259c640 1 -- 172.21.15.88:0/509239853 learned_addr learned my addr 172.21.15.88:0/509239853 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-29T10:42:04.349 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:04.358+0000 7f40c8845640 1 -- 172.21.15.88:0/509239853 --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40c40ac320 con 0x7f40c41457f0 2024-03-29T10:42:04.349 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:04.358+0000 7f40c8845640 1 -- 172.21.15.88:0/509239853 --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40c41420c0 con 0x7f40c413f650 2024-03-29T10:42:04.349 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:04.358+0000 7f40c8845640 1 -- 172.21.15.88:0/509239853 --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40c4137890 con 0x7f40c40a9d70 2024-03-29T10:42:04.349 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:04.358+0000 7f40b2ffd640 10 client.?.objecter ms_handle_connect 0x7f40c40a9d70 2024-03-29T10:42:04.349 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:04.358+0000 7f40b2ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:42:04.349 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:04.358+0000 7f40b2ffd640 10 client.?.objecter ms_handle_connect 0x7f40c413f650 2024-03-29T10:42:04.349 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:04.358+0000 7f40b2ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:42:04.350 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:04.358+0000 7f40c8845640 10 monclient(hunting): _renew_subs 2024-03-29T10:42:04.350 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:04.358+0000 7f40c8845640 10 monclient(hunting): authenticate will time out at 846.594490s 2024-03-29T10:42:04.350 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:04.358+0000 7f40b2ffd640 1 -- 172.21.15.88:0/509239853 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3987378919 0 0) 0x7f40c4137890 con 0x7f40c40a9d70 2024-03-29T10:42:04.350 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:04.358+0000 7f40b2ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:42:04.350 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:04.358+0000 7f40b2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:42:04.350 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:04.358+0000 7f40b2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:42:04.350 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:04.358+0000 7f40b2ffd640 10 monclient(hunting): my global_id is 4283 2024-03-29T10:42:04.350 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:04.358+0000 7f40b2ffd640 1 -- 172.21.15.88:0/509239853 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f40a4004eb0 con 0x7f40c40a9d70 2024-03-29T10:42:04.350 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:04.358+0000 7f40b2ffd640 1 -- 172.21.15.88:0/509239853 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3702130046 0 0) 0x7f40a4004eb0 con 0x7f40c40a9d70 2024-03-29T10:42:04.350 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:04.358+0000 7f40b2ffd640 1 -- 172.21.15.88:0/509239853 --> v1:172.21.15.88:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f40c4137890 con 0x7f40c40a9d70 2024-03-29T10:42:04.350 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:04.358+0000 7f40b2ffd640 1 -- 172.21.15.88:0/509239853 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f40b4003fe0 con 0x7f40c40a9d70 2024-03-29T10:42:04.350 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:04.358+0000 7f40b2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:04.350 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:04.358+0000 7f40b2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:42:04.350 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:04.358+0000 7f40b2ffd640 10 monclient(hunting): dump: 2024-03-29T10:42:04.351 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:42:04.351 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:42:04.351 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:42:04.351 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:42:04.351 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:04.351 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:42:04.351 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:42:04.351 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:42:04.351 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:42:04.351 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:42:04.351 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:04.358+0000 7f40b2ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:42:04.351 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:04.359+0000 7f40b2ffd640 1 -- 172.21.15.88:0/509239853 <== mon.2 v1:172.21.15.88:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2603307942 0 0) 0x7f40c4137890 con 0x7f40c40a9d70 2024-03-29T10:42:04.351 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:04.359+0000 7f40b2ffd640 1 -- 172.21.15.88:0/509239853 >> v1:172.21.15.196:6789/0 conn(0x7f40c413f650 legacy=0x7f40c41419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:04.351 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:04.359+0000 7f40b2ffd640 1 -- 172.21.15.88:0/509239853 >> v1:172.21.15.88:6789/0 conn(0x7f40c41457f0 legacy=0x7f40c4137180 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:42:04.352 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:04.359+0000 7f40b2ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:42:04.352 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:04.359+0000 7f40b2ffd640 1 monclient: found mon.c 2024-03-29T10:42:04.352 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:04.359+0000 7f40b2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:42:04.352 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:04.359+0000 7f40b2ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-03-29T10:42:04.352 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:04.359+0000 7f40b2ffd640 1 -- 172.21.15.88:0/509239853 --> v1:172.21.15.88:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40c4137a20 con 0x7f40c40a9d70 2024-03-29T10:42:04.352 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:04.359+0000 7f40b2ffd640 10 monclient: _finish_auth 0 2024-03-29T10:42:04.352 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:04.359+0000 7f40b2ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:42:04.352 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:04.359+0000 7f40b2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:42:04.352 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:04.359+0000 7f40c8845640 5 monclient: authenticate success, global_id 4283 2024-03-29T10:42:04.352 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:04.359+0000 7f40c8845640 10 monclient: _renew_subs 2024-03-29T10:42:04.352 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:04.359+0000 7f40c8845640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-03-29T10:42:04.352 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:04.360+0000 7f40c8845640 1 -- 172.21.15.88:0/509239853 --> v1:172.21.15.88:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40c4144840 con 0x7f40c40a9d70 2024-03-29T10:42:04.352 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:04.360+0000 7f40c8845640 10 client.4283.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:42:04.352 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:04.360+0000 7f40b2ffd640 1 -- 172.21.15.88:0/509239853 <== mon.2 v1:172.21.15.88:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f40b4000ed0 con 0x7f40c40a9d70 2024-03-29T10:42:04.353 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:04.360+0000 7f40b2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:42:04.353 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:04.360+0000 7f40c0d99640 4 set_mon_vals no callback set 2024-03-29T10:42:04.353 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:04.361+0000 7f40b2ffd640 1 -- 172.21.15.88:0/509239853 <== mon.2 v1:172.21.15.88:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f40b4001260 con 0x7f40c40a9d70 2024-03-29T10:42:04.353 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:04.361+0000 7f40b2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:04.353 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:04.361+0000 7f40b2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:42:04.353 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:04.361+0000 7f40b2ffd640 10 monclient: dump: 2024-03-29T10:42:04.353 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:42:04.353 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:42:04.353 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:42:04.353 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:42:04.353 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:04.353 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:42:04.353 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:42:04.353 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:42:04.353 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:42:04.354 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:42:04.354 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:04.361+0000 7f40c8845640 10 monclient: _renew_subs 2024-03-29T10:42:04.354 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:04.361+0000 7f40c8845640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-03-29T10:42:04.354 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:04.361+0000 7f40c8845640 1 -- 172.21.15.88:0/509239853 --> v1:172.21.15.88:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f40c4142bd0 con 0x7f40c40a9d70 2024-03-29T10:42:04.354 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:04.361+0000 7f40c8845640 1 librados: init done 2024-03-29T10:42:04.354 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:04.362+0000 7f40b2ffd640 1 -- 172.21.15.88:0/509239853 <== mon.2 v1:172.21.15.88:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4235629220 0 0) 0x7f40b4001440 con 0x7f40c40a9d70 2024-03-29T10:42:04.354 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:04.362+0000 7f40c8845640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:42:04.354 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:04.362+0000 7f40c8845640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:42:04.354 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:04.362+0000 7f40c8845640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-03-29T10:42:04.354 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:04.362+0000 7f40c8845640 1 -- 172.21.15.88:0/509239853 --> v1:172.21.15.88:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40c4142e80 con 0x7f40c40a9d70 2024-03-29T10:42:04.354 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:04.363+0000 7f40b2ffd640 1 -- 172.21.15.88:0/509239853 <== mon.2 v1:172.21.15.88:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4257673582 0 0) 0x7f40b4085bf0 con 0x7f40c40a9d70 2024-03-29T10:42:04.354 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:04.363+0000 7f40b2ffd640 10 client.4283.objecter ms_dispatch 0x7f40c4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:42:04.354 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:04.363+0000 7f40b2ffd640 3 client.4283.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:42:04.355 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:04.363+0000 7f40b2ffd640 3 client.4283.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:42:04.355 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:04.363+0000 7f40c1d9b640 10 monclient: get_auth_request con 0x7f40a4076390 auth_method 0 2024-03-29T10:42:04.355 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:04.363+0000 7f40b2ffd640 20 client.4283.objecter dump_active .. 0 homeless 2024-03-29T10:42:04.355 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:04.363+0000 7f40b2ffd640 10 client.4283.objecter ms_handle_connect 0x7f40a4076390 2024-03-29T10:42:04.355 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:04.369+0000 7f40b2ffd640 1 -- 172.21.15.88:0/509239853 <== mon.2 v1:172.21.15.88:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f40b40504a0 con 0x7f40c40a9d70 2024-03-29T10:42:04.355 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:04.369+0000 7f40b2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:42:04.355 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:04.369+0000 7f40b2ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:42:04.640 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:04.657+0000 7f40c8845640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:42:04.641 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:04.657+0000 7f40c8845640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:42:04.641 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:04.657+0000 7f40c8845640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-03-29T10:42:04.641 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:04.657+0000 7f40c8845640 1 -- 172.21.15.88:0/509239853 --> v1:172.21.15.88:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f40c4142250 con 0x7f40c40a9d70 2024-03-29T10:42:04.641 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:04.658+0000 7f40b2ffd640 1 -- 172.21.15.88:0/509239853 <== mon.2 v1:172.21.15.88:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 344506374) 0x7f40b40523d0 con 0x7f40c40a9d70 2024-03-29T10:42:04.641 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:04.658+0000 7f40b2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:42:04.641 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:04.658+0000 7f40b2ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:42:04.642 INFO:teuthology.misc.health.smithi088.stdout: 2024-03-29T10:42:04.642 INFO:teuthology.misc.health.smithi088.stdout:{"epoch":10,"fsid":"869f5188-f92b-42dd-a46d-8c646e0388a9","created":"2024-03-29T10:40:40.285087+0000","modified":"2024-03-29T10:40:42.536230+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:40:42.536230+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a26c0c1c-8de0-438c-a3f6-3df10ea61574","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5de24915-8dec-49b6-8b16-f3170197c591","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ac054af3-d589-4565-9644-e6af75975c53","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3ec4f045-c714-4a22-bd1a-e07330b45ab3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"cddb9867-8e30-4951-9b13-3d1e78b9c45a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dc6d9911-785d-432f-8b53-56a1ccab302f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"77c88437-b587-496e-8b4d-1824ea8b5305","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ce963d96-5a8e-44c3-99de-c0dbebe234ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:42:04.644 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:04.660+0000 7f40c8845640 10 librados: watch_flush enter 2024-03-29T10:42:04.644 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:04.660+0000 7f40c8845640 10 librados: watch_flush exit 2024-03-29T10:42:04.644 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:04.660+0000 7f40c8845640 20 client.4283.objecter shutdown clearing up homeless session... 2024-03-29T10:42:04.644 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:04.660+0000 7f40c8845640 10 client.4283.objecter successfully canceled tick 2024-03-29T10:42:04.645 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:04.661+0000 7f40c8845640 1 -- 172.21.15.88:0/509239853 >> v1:172.21.15.88:6816/1752861349 conn(0x7f40a4076390 legacy=0x7f40a4078850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:04.645 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:04.661+0000 7f40c8845640 10 monclient: shutdown 2024-03-29T10:42:04.645 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:04.661+0000 7f40c8845640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:42:04.645 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:04.661+0000 7f40c8845640 1 -- 172.21.15.88:0/509239853 >> v1:172.21.15.88:6790/0 conn(0x7f40c40a9d70 legacy=0x7f40c413fda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:04.645 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:04.661+0000 7f40c8845640 1 -- 172.21.15.88:0/509239853 shutdown_connections 2024-03-29T10:42:04.645 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:04.661+0000 7f40c8845640 1 -- 172.21.15.88:0/509239853 >> 172.21.15.88:0/509239853 conn(0x7f40c4066a00 msgr2=0x7f40c413a970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:42:04.645 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:04.661+0000 7f40c8845640 1 -- 172.21.15.88:0/509239853 shutdown_connections 2024-03-29T10:42:04.645 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:04.661+0000 7f40c8845640 1 -- 172.21.15.88:0/509239853 wait complete. 2024-03-29T10:42:04.645 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:04.661+0000 7f40c8845640 1 librados: shutdown 2024-03-29T10:42:04.657 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:42:11.464 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:42:11.600 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:11.617+0000 7f5c5bed9640 10 monclient: get_monmap_and_config 2024-03-29T10:42:11.601 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:11.617+0000 7f5c5bed9640 10 monclient: build_initial_monmap 2024-03-29T10:42:11.601 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:11.617+0000 7f5c5bed9640 10 monclient: monmap: 2024-03-29T10:42:11.601 INFO:teuthology.misc.health.smithi088.stderr:epoch 0 2024-03-29T10:42:11.601 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:42:11.601 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:42:11.618333+0000 2024-03-29T10:42:11.601 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:42:11.618333+0000 2024-03-29T10:42:11.601 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 0 (unknown) 2024-03-29T10:42:11.601 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:42:11.601 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-03-29T10:42:11.601 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-29T10:42:11.601 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-b 2024-03-29T10:42:11.601 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:42:11.602 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:11.618+0000 7f5c5bed9640 1 Processor -- start 2024-03-29T10:42:11.602 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:11.618+0000 7f5c5bed9640 1 -- start start 2024-03-29T10:42:11.602 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:11.618+0000 7f5c5bed9640 10 monclient: init 2024-03-29T10:42:11.603 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:11.619+0000 7f5c5bed9640 10 monclient: _reopen_session rank -1 2024-03-29T10:42:11.603 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:11.619+0000 7f5c5bed9640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:42:11.603 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:11.619+0000 7f5c5bed9640 10 monclient(hunting): picked mon.noname-a con 0x7f5c541457f0 addr v1:172.21.15.88:6789/0 2024-03-29T10:42:11.603 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:11.619+0000 7f5c5bed9640 10 monclient(hunting): picked mon.noname-c con 0x7f5c5413f650 addr v1:172.21.15.196:6789/0 2024-03-29T10:42:11.603 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:11.619+0000 7f5c5bed9640 10 monclient(hunting): picked mon.noname-b con 0x7f5c540a9d70 addr v1:172.21.15.88:6790/0 2024-03-29T10:42:11.603 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:11.619+0000 7f5c5bed9640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c540aaef0 con 0x7f5c541457f0 2024-03-29T10:42:11.603 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:11.619+0000 7f5c5bed9640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c54145bd0 con 0x7f5c5413f650 2024-03-29T10:42:11.603 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:11.619+0000 7f5c5bed9640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c54145eb0 con 0x7f5c540a9d70 2024-03-29T10:42:11.603 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:11.619+0000 7f5c5bed9640 10 monclient(hunting): _renew_subs 2024-03-29T10:42:11.603 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:11.619+0000 7f5c5bed9640 10 monclient(hunting): authenticate will time out at 853.855385s 2024-03-29T10:42:11.605 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:11.621+0000 7f5c5a476640 1 --1- >> v1:172.21.15.88:6790/0 conn(0x7f5c540a9d70 0x7f5c5414d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6790/0 says I am v1:172.21.15.88:60468/0 (socket says 172.21.15.88:60468) 2024-03-29T10:42:11.605 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:11.621+0000 7f5c5a476640 1 -- 172.21.15.88:0/1353460167 learned_addr learned my addr 172.21.15.88:0/1353460167 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-29T10:42:11.607 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:11.623+0000 7f5c58c73640 1 -- 172.21.15.88:0/1353460167 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2730607502 0 0) 0x7f5c54145eb0 con 0x7f5c540a9d70 2024-03-29T10:42:11.607 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:11.624+0000 7f5c58c73640 10 monclient(hunting): _negotiate 2024-03-29T10:42:11.607 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:11.624+0000 7f5c58c73640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:42:11.607 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:11.624+0000 7f5c58c73640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:42:11.607 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:11.624+0000 7f5c58c73640 10 monclient(hunting): my global_id is 4286 2024-03-29T10:42:11.607 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:11.624+0000 7f5c58c73640 1 -- 172.21.15.88:0/1353460167 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5c40003520 con 0x7f5c540a9d70 2024-03-29T10:42:11.607 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:11.624+0000 7f5c58c73640 1 -- 172.21.15.88:0/1353460167 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3668983747 0 0) 0x7f5c540aaef0 con 0x7f5c541457f0 2024-03-29T10:42:11.608 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:11.624+0000 7f5c58c73640 10 monclient(hunting): _negotiate 2024-03-29T10:42:11.608 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:11.624+0000 7f5c58c73640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:42:11.608 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:11.624+0000 7f5c58c73640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:42:11.608 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:11.624+0000 7f5c58c73640 10 monclient(hunting): my global_id is 4287 2024-03-29T10:42:11.608 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:11.624+0000 7f5c58c73640 1 -- 172.21.15.88:0/1353460167 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5c54145eb0 con 0x7f5c541457f0 2024-03-29T10:42:11.608 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:11.624+0000 7f5c58c73640 1 -- 172.21.15.88:0/1353460167 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4260506631 0 0) 0x7f5c54145eb0 con 0x7f5c541457f0 2024-03-29T10:42:11.608 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:11.624+0000 7f5c58c73640 1 -- 172.21.15.88:0/1353460167 >> v1:172.21.15.88:6790/0 conn(0x7f5c540a9d70 legacy=0x7f5c5414d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:11.608 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:11.624+0000 7f5c58c73640 1 -- 172.21.15.88:0/1353460167 >> v1:172.21.15.196:6789/0 conn(0x7f5c5413f650 legacy=0x7f5c540b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:11.611 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:11.624+0000 7f5c58c73640 10 monclient: _finish_hunting 0 2024-03-29T10:42:11.611 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:11.624+0000 7f5c58c73640 1 monclient: found mon.noname-a 2024-03-29T10:42:11.611 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:11.624+0000 7f5c58c73640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:42:11.611 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:11.624+0000 7f5c58c73640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.88:6789/0 2024-03-29T10:42:11.611 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:11.624+0000 7f5c58c73640 1 -- 172.21.15.88:0/1353460167 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c54146340 con 0x7f5c541457f0 2024-03-29T10:42:11.611 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:11.624+0000 7f5c58c73640 10 monclient: _finish_auth 0 2024-03-29T10:42:11.611 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:11.624+0000 7f5c58c73640 10 monclient: _check_auth_tickets 2024-03-29T10:42:11.612 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:11.624+0000 7f5c58c73640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:42:11.612 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:11.624+0000 7f5c58c73640 1 -- 172.21.15.88:0/1353460167 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f5c4c001970 con 0x7f5c541457f0 2024-03-29T10:42:11.612 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:11.624+0000 7f5c58c73640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:11.612 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:11.624+0000 7f5c58c73640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:42:11.612 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:11.624+0000 7f5c58c73640 10 monclient: dump: 2024-03-29T10:42:11.612 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:42:11.612 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:42:11.612 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:42:11.612 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:42:11.612 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:11.612 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:42:11.612 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:42:11.612 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:42:11.613 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:42:11.613 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:42:11.613 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:11.624+0000 7f5c5bed9640 5 monclient: authenticate success, global_id 4287 2024-03-29T10:42:11.613 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:11.624+0000 7f5c5bed9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:42:11.613 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:11.624+0000 7f5c58c73640 1 -- 172.21.15.88:0/1353460167 <== mon.0 v1:172.21.15.88:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5c4c001b90 con 0x7f5c541457f0 2024-03-29T10:42:11.613 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:11.624+0000 7f5c58c73640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:42:11.613 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:11.624+0000 7f5c58c73640 1 -- 172.21.15.88:0/1353460167 <== mon.0 v1:172.21.15.88:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f5c4c001f20 con 0x7f5c541457f0 2024-03-29T10:42:11.613 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:11.624+0000 7f5c58c73640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:11.613 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:11.624+0000 7f5c58c73640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:42:11.613 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:11.624+0000 7f5c58c73640 10 monclient: dump: 2024-03-29T10:42:11.613 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:42:11.613 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:42:11.613 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:42:11.613 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:42:11.614 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:11.614 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:42:11.614 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:42:11.614 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:42:11.614 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:42:11.614 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:42:11.614 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:11.624+0000 7f5c5bed9640 10 monclient: get_monmap_and_config success 2024-03-29T10:42:11.614 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:11.624+0000 7f5c5bed9640 4 set_mon_vals no callback set 2024-03-29T10:42:11.614 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:11.625+0000 7f5c5bed9640 10 monclient: shutdown 2024-03-29T10:42:11.614 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:11.625+0000 7f5c5bed9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:42:11.614 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:11.625+0000 7f5c5bed9640 1 -- 172.21.15.88:0/1353460167 >> v1:172.21.15.88:6789/0 conn(0x7f5c541457f0 legacy=0x7f5c5413ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:11.614 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:11.625+0000 7f5c5bed9640 1 -- 172.21.15.88:0/1353460167 shutdown_connections 2024-03-29T10:42:11.614 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:11.625+0000 7f5c5bed9640 1 -- 172.21.15.88:0/1353460167 >> 172.21.15.88:0/1353460167 conn(0x7f5c54066a00 msgr2=0x7f5c54066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:42:11.614 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:11.625+0000 7f5c5bed9640 1 -- 172.21.15.88:0/1353460167 shutdown_connections 2024-03-29T10:42:11.615 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:11.625+0000 7f5c5bed9640 1 -- 172.21.15.88:0/1353460167 wait complete. 2024-03-29T10:42:11.615 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:11.627+0000 7f5c5bed9640 10 monclient: build_initial_monmap 2024-03-29T10:42:11.615 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:11.627+0000 7f5c5bed9640 10 monclient: monmap: 2024-03-29T10:42:11.615 INFO:teuthology.misc.health.smithi088.stderr:epoch 0 2024-03-29T10:42:11.615 INFO:teuthology.misc.health.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:42:11.615 INFO:teuthology.misc.health.smithi088.stderr:last_changed 0.000000 2024-03-29T10:42:11.615 INFO:teuthology.misc.health.smithi088.stderr:created 0.000000 2024-03-29T10:42:11.615 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 0 (unknown) 2024-03-29T10:42:11.615 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:42:11.615 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-03-29T10:42:11.615 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-29T10:42:11.615 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-c 2024-03-29T10:42:11.615 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:42:11.615 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:11.627+0000 7f5c5bed9640 1 librados: starting msgr at 2024-03-29T10:42:11.615 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:11.627+0000 7f5c5bed9640 1 librados: starting objecter 2024-03-29T10:42:11.616 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:11.627+0000 7f5c5bed9640 1 Processor -- start 2024-03-29T10:42:11.619 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:11.628+0000 7f5c5bed9640 1 -- start start 2024-03-29T10:42:11.619 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:11.628+0000 7f5c5bed9640 1 librados: setting wanted keys 2024-03-29T10:42:11.619 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:11.628+0000 7f5c5bed9640 1 librados: calling monclient init 2024-03-29T10:42:11.619 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:11.628+0000 7f5c5bed9640 10 monclient: init 2024-03-29T10:42:11.619 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:11.628+0000 7f5c5bed9640 10 monclient: _reopen_session rank -1 2024-03-29T10:42:11.620 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:11.628+0000 7f5c5bed9640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:42:11.620 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:11.628+0000 7f5c5bed9640 10 monclient(hunting): picked mon.noname-a con 0x7f5c540a9d70 addr v1:172.21.15.88:6789/0 2024-03-29T10:42:11.620 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:11.628+0000 7f5c5bed9640 10 monclient(hunting): picked mon.noname-b con 0x7f5c5413f650 addr v1:172.21.15.196:6789/0 2024-03-29T10:42:11.620 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:11.628+0000 7f5c5bed9640 10 monclient(hunting): picked mon.noname-c con 0x7f5c541457f0 addr v1:172.21.15.88:6790/0 2024-03-29T10:42:11.620 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:11.629+0000 7f5c5bed9640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c5406ef60 con 0x7f5c540a9d70 2024-03-29T10:42:11.620 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:11.629+0000 7f5c5bed9640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c54070ac0 con 0x7f5c5413f650 2024-03-29T10:42:11.620 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:11.629+0000 7f5c5bed9640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c54070da0 con 0x7f5c541457f0 2024-03-29T10:42:11.620 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:11.629+0000 7f5c5bed9640 10 monclient(hunting): _renew_subs 2024-03-29T10:42:11.620 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:11.629+0000 7f5c5bed9640 10 monclient(hunting): authenticate will time out at 853.865493s 2024-03-29T10:42:11.620 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:11.629+0000 7f5c5a476640 1 --1- >> v1:172.21.15.88:6790/0 conn(0x7f5c541457f0 0x7f5c5406fb00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6790/0 says I am v1:172.21.15.88:41224/0 (socket says 172.21.15.88:41224) 2024-03-29T10:42:11.620 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:11.629+0000 7f5c5a476640 1 -- 172.21.15.88:0/2261231214 learned_addr learned my addr 172.21.15.88:0/2261231214 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-29T10:42:11.620 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:11.629+0000 7f5c4a7fc640 10 client.?.objecter ms_handle_connect 0x7f5c541457f0 2024-03-29T10:42:11.620 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:11.629+0000 7f5c4a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:42:11.621 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:11.629+0000 7f5c4a7fc640 1 -- 172.21.15.88:0/2261231214 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3204863473 0 0) 0x7f5c54070da0 con 0x7f5c541457f0 2024-03-29T10:42:11.621 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:11.630+0000 7f5c4a7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:42:11.621 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:11.630+0000 7f5c4a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:42:11.621 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:11.630+0000 7f5c4a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:42:11.621 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:11.630+0000 7f5c4a7fc640 10 monclient(hunting): my global_id is 4289 2024-03-29T10:42:11.621 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:11.630+0000 7f5c4a7fc640 1 -- 172.21.15.88:0/2261231214 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5c38004eb0 con 0x7f5c541457f0 2024-03-29T10:42:11.621 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:11.630+0000 7f5c4a7fc640 1 -- 172.21.15.88:0/2261231214 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1271554791 0 0) 0x7f5c38004eb0 con 0x7f5c541457f0 2024-03-29T10:42:11.621 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:11.630+0000 7f5c4a7fc640 1 -- 172.21.15.88:0/2261231214 --> v1:172.21.15.88:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5c54070da0 con 0x7f5c541457f0 2024-03-29T10:42:11.621 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:11.630+0000 7f5c4a7fc640 1 -- 172.21.15.88:0/2261231214 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f5c500019a0 con 0x7f5c541457f0 2024-03-29T10:42:11.621 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:11.630+0000 7f5c4a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:11.621 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:11.630+0000 7f5c4a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:42:11.621 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:11.630+0000 7f5c4a7fc640 10 monclient(hunting): dump: 2024-03-29T10:42:11.621 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:42:11.621 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:42:11.622 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:42:11.622 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:42:11.622 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:11.622 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:42:11.622 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:42:11.622 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:42:11.622 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:42:11.622 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:42:11.622 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:11.630+0000 7f5c4a7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:42:11.622 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:11.630+0000 7f5c4a7fc640 1 -- 172.21.15.88:0/2261231214 <== mon.2 v1:172.21.15.88:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 892559449 0 0) 0x7f5c54070da0 con 0x7f5c541457f0 2024-03-29T10:42:11.622 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:11.630+0000 7f5c4a7fc640 1 -- 172.21.15.88:0/2261231214 >> v1:172.21.15.196:6789/0 conn(0x7f5c5413f650 legacy=0x7f5c5406f3f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:42:11.622 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:11.632+0000 7f5c4a7fc640 1 -- 172.21.15.88:0/2261231214 >> v1:172.21.15.88:6789/0 conn(0x7f5c540a9d70 legacy=0x7f5c54071810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:11.622 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:11.632+0000 7f5c4a7fc640 10 monclient: _finish_hunting 0 2024-03-29T10:42:11.623 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:11.632+0000 7f5c4a7fc640 1 monclient: found mon.c 2024-03-29T10:42:11.623 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:11.632+0000 7f5c4a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:42:11.623 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:11.632+0000 7f5c4a7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-03-29T10:42:11.623 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:11.632+0000 7f5c4a7fc640 1 -- 172.21.15.88:0/2261231214 --> v1:172.21.15.88:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c5406e9b0 con 0x7f5c541457f0 2024-03-29T10:42:11.623 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:11.632+0000 7f5c4a7fc640 10 monclient: _finish_auth 0 2024-03-29T10:42:11.623 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:11.632+0000 7f5c4a7fc640 10 monclient: _check_auth_tickets 2024-03-29T10:42:11.623 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:11.632+0000 7f5c4a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:42:11.623 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:11.632+0000 7f5c5bed9640 5 monclient: authenticate success, global_id 4289 2024-03-29T10:42:11.623 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:11.632+0000 7f5c5bed9640 10 monclient: _renew_subs 2024-03-29T10:42:11.623 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:11.632+0000 7f5c5bed9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-03-29T10:42:11.623 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:11.632+0000 7f5c5bed9640 1 -- 172.21.15.88:0/2261231214 --> v1:172.21.15.88:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c5406ec60 con 0x7f5c541457f0 2024-03-29T10:42:11.623 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:11.632+0000 7f5c5bed9640 10 client.4289.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:42:11.623 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:11.632+0000 7f5c4a7fc640 1 -- 172.21.15.88:0/2261231214 <== mon.2 v1:172.21.15.88:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5c50001b00 con 0x7f5c541457f0 2024-03-29T10:42:11.623 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:11.632+0000 7f5c4a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:42:11.623 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:11.632+0000 7f5c4bfff640 4 set_mon_vals no callback set 2024-03-29T10:42:11.624 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:11.632+0000 7f5c5bed9640 10 monclient: _renew_subs 2024-03-29T10:42:11.624 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:11.632+0000 7f5c5bed9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-03-29T10:42:11.624 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:11.632+0000 7f5c5bed9640 1 -- 172.21.15.88:0/2261231214 --> v1:172.21.15.88:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c541e7280 con 0x7f5c541457f0 2024-03-29T10:42:11.624 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:11.632+0000 7f5c5bed9640 1 librados: init done 2024-03-29T10:42:11.624 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:11.632+0000 7f5c4a7fc640 1 -- 172.21.15.88:0/2261231214 <== mon.2 v1:172.21.15.88:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f5c50004a90 con 0x7f5c541457f0 2024-03-29T10:42:11.624 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:11.633+0000 7f5c4a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:11.624 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:11.633+0000 7f5c4a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:42:11.624 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:11.633+0000 7f5c4a7fc640 10 monclient: dump: 2024-03-29T10:42:11.624 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:42:11.624 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:42:11.624 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:42:11.624 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:42:11.624 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:11.624 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:42:11.625 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:42:11.625 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:42:11.625 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:42:11.625 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:42:11.625 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:11.634+0000 7f5c4a7fc640 1 -- 172.21.15.88:0/2261231214 <== mon.2 v1:172.21.15.88:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4235629220 0 0) 0x7f5c5001aa60 con 0x7f5c541457f0 2024-03-29T10:42:11.625 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:11.635+0000 7f5c4a7fc640 1 -- 172.21.15.88:0/2261231214 <== mon.2 v1:172.21.15.88:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4257673582 0 0) 0x7f5c50084b30 con 0x7f5c541457f0 2024-03-29T10:42:11.625 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:11.635+0000 7f5c4a7fc640 10 client.4289.objecter ms_dispatch 0x7f5c54000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:42:11.625 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:11.635+0000 7f5c4a7fc640 3 client.4289.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:42:11.625 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:11.635+0000 7f5c4a7fc640 3 client.4289.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:42:11.625 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:11.635+0000 7f5c4a7fc640 20 client.4289.objecter dump_active .. 0 homeless 2024-03-29T10:42:11.625 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:11.635+0000 7f5c5bed9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:42:11.625 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:11.635+0000 7f5c5bed9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:42:11.625 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:11.635+0000 7f5c5bed9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-03-29T10:42:11.625 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:11.635+0000 7f5c5bed9640 1 -- 172.21.15.88:0/2261231214 --> v1:172.21.15.88:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c54075f10 con 0x7f5c541457f0 2024-03-29T10:42:11.625 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:11.635+0000 7f5c59c75640 10 monclient: get_auth_request con 0x7f5c3806dcd0 auth_method 0 2024-03-29T10:42:11.626 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:11.637+0000 7f5c4a7fc640 10 client.4289.objecter ms_handle_connect 0x7f5c3806dcd0 2024-03-29T10:42:11.627 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:11.644+0000 7f5c4a7fc640 1 -- 172.21.15.88:0/2261231214 <== mon.2 v1:172.21.15.88:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5c50050240 con 0x7f5c541457f0 2024-03-29T10:42:11.627 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:11.644+0000 7f5c4a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:42:11.627 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:11.644+0000 7f5c4a7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:42:11.928 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:11.944+0000 7f5c5bed9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:42:11.928 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:11.944+0000 7f5c5bed9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:42:11.928 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:11.944+0000 7f5c5bed9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-03-29T10:42:11.928 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:11.944+0000 7f5c5bed9640 1 -- 172.21.15.88:0/2261231214 --> v1:172.21.15.88:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5c5406f110 con 0x7f5c541457f0 2024-03-29T10:42:11.928 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:11.945+0000 7f5c4a7fc640 1 -- 172.21.15.88:0/2261231214 <== mon.2 v1:172.21.15.88:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 344506374) 0x7f5c50052340 con 0x7f5c541457f0 2024-03-29T10:42:11.929 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:11.945+0000 7f5c4a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:42:11.929 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:11.945+0000 7f5c4a7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:42:11.929 INFO:teuthology.misc.health.smithi088.stdout: 2024-03-29T10:42:11.929 INFO:teuthology.misc.health.smithi088.stdout:{"epoch":10,"fsid":"869f5188-f92b-42dd-a46d-8c646e0388a9","created":"2024-03-29T10:40:40.285087+0000","modified":"2024-03-29T10:40:42.536230+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:40:42.536230+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a26c0c1c-8de0-438c-a3f6-3df10ea61574","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5de24915-8dec-49b6-8b16-f3170197c591","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ac054af3-d589-4565-9644-e6af75975c53","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3ec4f045-c714-4a22-bd1a-e07330b45ab3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"cddb9867-8e30-4951-9b13-3d1e78b9c45a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dc6d9911-785d-432f-8b53-56a1ccab302f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"77c88437-b587-496e-8b4d-1824ea8b5305","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ce963d96-5a8e-44c3-99de-c0dbebe234ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:42:11.932 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:11.948+0000 7f5c5bed9640 10 librados: watch_flush enter 2024-03-29T10:42:11.932 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:11.948+0000 7f5c5bed9640 10 librados: watch_flush exit 2024-03-29T10:42:11.932 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:11.948+0000 7f5c5bed9640 20 client.4289.objecter shutdown clearing up homeless session... 2024-03-29T10:42:11.933 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:11.948+0000 7f5c5bed9640 10 client.4289.objecter successfully canceled tick 2024-03-29T10:42:11.933 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:11.948+0000 7f5c5bed9640 1 -- 172.21.15.88:0/2261231214 >> v1:172.21.15.88:6816/1752861349 conn(0x7f5c3806dcd0 legacy=0x7f5c38070190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:11.933 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:11.948+0000 7f5c5bed9640 10 monclient: shutdown 2024-03-29T10:42:11.933 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:11.948+0000 7f5c5bed9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:42:11.933 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:11.948+0000 7f5c5bed9640 1 -- 172.21.15.88:0/2261231214 >> v1:172.21.15.88:6790/0 conn(0x7f5c541457f0 legacy=0x7f5c5406fb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:11.933 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:11.949+0000 7f5c5bed9640 1 -- 172.21.15.88:0/2261231214 shutdown_connections 2024-03-29T10:42:11.933 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:11.949+0000 7f5c5bed9640 1 -- 172.21.15.88:0/2261231214 >> 172.21.15.88:0/2261231214 conn(0x7f5c54066a00 msgr2=0x7f5c54066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:42:11.933 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:11.949+0000 7f5c5bed9640 1 -- 172.21.15.88:0/2261231214 shutdown_connections 2024-03-29T10:42:11.933 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:11.949+0000 7f5c5bed9640 1 -- 172.21.15.88:0/2261231214 wait complete. 2024-03-29T10:42:11.933 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:11.949+0000 7f5c5bed9640 1 librados: shutdown 2024-03-29T10:42:11.946 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:42:18.753 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:42:18.909 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:18.925+0000 7f3eeb84e640 10 monclient: get_monmap_and_config 2024-03-29T10:42:18.909 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:18.925+0000 7f3eeb84e640 10 monclient: build_initial_monmap 2024-03-29T10:42:18.909 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:18.925+0000 7f3eeb84e640 10 monclient: monmap: 2024-03-29T10:42:18.909 INFO:teuthology.misc.health.smithi088.stderr:epoch 0 2024-03-29T10:42:18.909 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:42:18.909 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:42:18.926723+0000 2024-03-29T10:42:18.909 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:42:18.926723+0000 2024-03-29T10:42:18.910 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 0 (unknown) 2024-03-29T10:42:18.910 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:42:18.910 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-03-29T10:42:18.910 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-29T10:42:18.910 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-b 2024-03-29T10:42:18.910 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:42:18.911 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:18.927+0000 7f3eeb84e640 1 Processor -- start 2024-03-29T10:42:18.911 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:18.927+0000 7f3eeb84e640 1 -- start start 2024-03-29T10:42:18.911 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:18.927+0000 7f3eeb84e640 10 monclient: init 2024-03-29T10:42:18.912 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:18.928+0000 7f3eeb84e640 10 monclient: _reopen_session rank -1 2024-03-29T10:42:18.912 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:18.928+0000 7f3eeb84e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:42:18.913 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:18.928+0000 7f3eeb84e640 10 monclient(hunting): picked mon.noname-c con 0x7f3ee40b0500 addr v1:172.21.15.196:6789/0 2024-03-29T10:42:18.913 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:18.928+0000 7f3eeb84e640 10 monclient(hunting): picked mon.noname-a con 0x7f3ee40b08e0 addr v1:172.21.15.88:6789/0 2024-03-29T10:42:18.913 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:18.928+0000 7f3eeb84e640 10 monclient(hunting): picked mon.noname-b con 0x7f3ee40aeca0 addr v1:172.21.15.88:6790/0 2024-03-29T10:42:18.913 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:18.928+0000 7f3eeb84e640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ee4140620 con 0x7f3ee40b08e0 2024-03-29T10:42:18.913 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:18.928+0000 7f3eeb84e640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ee413daf0 con 0x7f3ee40b0500 2024-03-29T10:42:18.913 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:18.928+0000 7f3eeb84e640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ee40ae180 con 0x7f3ee40aeca0 2024-03-29T10:42:18.913 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:18.928+0000 7f3eeb84e640 10 monclient(hunting): _renew_subs 2024-03-29T10:42:18.913 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:18.928+0000 7f3eeb84e640 10 monclient(hunting): authenticate will time out at 861.164423s 2024-03-29T10:42:18.916 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:18.933+0000 7f3ee95ea640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f3ee40b0500 0x7f3ee413ff10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.88:38113/0 (socket says 172.21.15.88:38113) 2024-03-29T10:42:18.916 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:18.933+0000 7f3ee95ea640 1 -- 172.21.15.88:0/1301298532 learned_addr learned my addr 172.21.15.88:0/1301298532 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-29T10:42:18.922 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:18.934+0000 7f3ee3fff640 1 -- 172.21.15.88:0/1301298532 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 128245404 0 0) 0x7f3ee413daf0 con 0x7f3ee40b0500 2024-03-29T10:42:18.922 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:18.934+0000 7f3ee3fff640 10 monclient(hunting): _negotiate 2024-03-29T10:42:18.922 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:18.934+0000 7f3ee3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:42:18.922 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:18.934+0000 7f3ee3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:42:18.922 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:18.934+0000 7f3ee3fff640 10 monclient(hunting): my global_id is 4276 2024-03-29T10:42:18.922 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:18.934+0000 7f3ee3fff640 1 -- 172.21.15.88:0/1301298532 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3ed4003520 con 0x7f3ee40b0500 2024-03-29T10:42:18.922 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:18.935+0000 7f3ee3fff640 1 -- 172.21.15.88:0/1301298532 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 478221448 0 0) 0x7f3ed4003520 con 0x7f3ee40b0500 2024-03-29T10:42:18.922 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:18.935+0000 7f3ee3fff640 1 -- 172.21.15.88:0/1301298532 >> v1:172.21.15.88:6790/0 conn(0x7f3ee40aeca0 legacy=0x7f3ee414b7d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:42:18.922 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:18.935+0000 7f3ee3fff640 1 -- 172.21.15.88:0/1301298532 >> v1:172.21.15.88:6789/0 conn(0x7f3ee40b08e0 legacy=0x7f3ee413d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:18.923 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:18.935+0000 7f3ee3fff640 10 monclient: _finish_hunting 0 2024-03-29T10:42:18.923 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:18.935+0000 7f3ee3fff640 1 monclient: found mon.noname-c 2024-03-29T10:42:18.923 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:18.935+0000 7f3ee3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:42:18.923 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:18.935+0000 7f3ee3fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.196:6789/0 2024-03-29T10:42:18.923 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:18.935+0000 7f3ee3fff640 1 -- 172.21.15.88:0/1301298532 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ee40accf0 con 0x7f3ee40b0500 2024-03-29T10:42:18.923 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:18.935+0000 7f3ee3fff640 10 monclient: _finish_auth 0 2024-03-29T10:42:18.923 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:18.935+0000 7f3ee3fff640 10 monclient: _check_auth_tickets 2024-03-29T10:42:18.923 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:18.936+0000 7f3ee3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:42:18.923 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:18.936+0000 7f3eeb84e640 5 monclient: authenticate success, global_id 4276 2024-03-29T10:42:18.923 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:18.936+0000 7f3eeb84e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:42:18.923 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:18.936+0000 7f3ee3fff640 1 -- 172.21.15.88:0/1301298532 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f3ed0001690 con 0x7f3ee40b0500 2024-03-29T10:42:18.923 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:18.936+0000 7f3ee3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:18.923 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:18.936+0000 7f3ee3fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:42:18.923 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:18.936+0000 7f3ee3fff640 10 monclient: dump: 2024-03-29T10:42:18.924 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:42:18.924 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:42:18.924 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:42:18.924 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:42:18.924 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:18.924 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:42:18.924 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:42:18.924 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:42:18.924 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:42:18.924 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:42:18.924 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:18.936+0000 7f3eeb84e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:42:18.924 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:18.936+0000 7f3ee3fff640 1 -- 172.21.15.88:0/1301298532 <== mon.1 v1:172.21.15.196:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3ed0001920 con 0x7f3ee40b0500 2024-03-29T10:42:18.924 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:18.936+0000 7f3ee3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:42:18.924 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:18.936+0000 7f3eeb84e640 10 monclient: get_monmap_and_config success 2024-03-29T10:42:18.925 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:18.936+0000 7f3eeb84e640 4 set_mon_vals no callback set 2024-03-29T10:42:18.925 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:18.937+0000 7f3eeb84e640 10 monclient: shutdown 2024-03-29T10:42:18.925 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:18.937+0000 7f3eeb84e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:42:18.925 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:18.937+0000 7f3eeb84e640 1 -- 172.21.15.88:0/1301298532 >> v1:172.21.15.196:6789/0 conn(0x7f3ee40b0500 legacy=0x7f3ee413ff10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:18.925 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:18.937+0000 7f3eeb84e640 1 -- 172.21.15.88:0/1301298532 shutdown_connections 2024-03-29T10:42:18.925 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:18.937+0000 7f3eeb84e640 1 -- 172.21.15.88:0/1301298532 >> 172.21.15.88:0/1301298532 conn(0x7f3ee406b660 msgr2=0x7f3ee406ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:42:18.925 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:18.938+0000 7f3eeb84e640 1 -- 172.21.15.88:0/1301298532 shutdown_connections 2024-03-29T10:42:18.929 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:18.938+0000 7f3eeb84e640 1 -- 172.21.15.88:0/1301298532 wait complete. 2024-03-29T10:42:18.929 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:18.938+0000 7f3eeb84e640 10 monclient: build_initial_monmap 2024-03-29T10:42:18.929 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:18.938+0000 7f3eeb84e640 10 monclient: monmap: 2024-03-29T10:42:18.929 INFO:teuthology.misc.health.smithi088.stderr:epoch 0 2024-03-29T10:42:18.929 INFO:teuthology.misc.health.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:42:18.929 INFO:teuthology.misc.health.smithi088.stderr:last_changed 0.000000 2024-03-29T10:42:18.929 INFO:teuthology.misc.health.smithi088.stderr:created 0.000000 2024-03-29T10:42:18.929 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 0 (unknown) 2024-03-29T10:42:18.929 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:42:18.929 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-03-29T10:42:18.930 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-29T10:42:18.930 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-c 2024-03-29T10:42:18.930 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:42:18.930 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:18.939+0000 7f3eeb84e640 1 librados: starting msgr at 2024-03-29T10:42:18.930 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:18.939+0000 7f3eeb84e640 1 librados: starting objecter 2024-03-29T10:42:18.930 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:18.939+0000 7f3eeb84e640 1 Processor -- start 2024-03-29T10:42:18.930 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:18.939+0000 7f3eeb84e640 1 -- start start 2024-03-29T10:42:18.930 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:18.939+0000 7f3eeb84e640 1 librados: setting wanted keys 2024-03-29T10:42:18.930 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:18.939+0000 7f3eeb84e640 1 librados: calling monclient init 2024-03-29T10:42:18.930 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:18.939+0000 7f3eeb84e640 10 monclient: init 2024-03-29T10:42:18.930 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:18.940+0000 7f3eeb84e640 10 monclient: _reopen_session rank -1 2024-03-29T10:42:18.930 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:18.940+0000 7f3eeb84e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:42:18.930 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:18.940+0000 7f3eeb84e640 10 monclient(hunting): picked mon.noname-a con 0x7f3ee40aeca0 addr v1:172.21.15.88:6789/0 2024-03-29T10:42:18.931 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:18.940+0000 7f3eeb84e640 10 monclient(hunting): picked mon.noname-b con 0x7f3ee40b0500 addr v1:172.21.15.196:6789/0 2024-03-29T10:42:18.931 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:18.940+0000 7f3eeb84e640 10 monclient(hunting): picked mon.noname-c con 0x7f3ee41cc710 addr v1:172.21.15.88:6790/0 2024-03-29T10:42:18.931 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:18.940+0000 7f3eeb84e640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ee41cd330 con 0x7f3ee40aeca0 2024-03-29T10:42:18.931 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:18.940+0000 7f3eeb84e640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ee41e1430 con 0x7f3ee40b0500 2024-03-29T10:42:18.931 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:18.940+0000 7f3eeb84e640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ee41e15c0 con 0x7f3ee41cc710 2024-03-29T10:42:18.931 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:18.940+0000 7f3eeb84e640 10 monclient(hunting): _renew_subs 2024-03-29T10:42:18.931 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:18.940+0000 7f3eeb84e640 10 monclient(hunting): authenticate will time out at 861.176497s 2024-03-29T10:42:18.931 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:18.944+0000 7f3ee8de9640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f3ee40b0500 0x7f3ee41e0d20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.88:38175/0 (socket says 172.21.15.88:38175) 2024-03-29T10:42:18.931 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:18.944+0000 7f3ee8de9640 1 -- 172.21.15.88:0/1965152841 learned_addr learned my addr 172.21.15.88:0/1965152841 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-29T10:42:18.931 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:18.944+0000 7f3ee1ffb640 10 client.?.objecter ms_handle_connect 0x7f3ee40b0500 2024-03-29T10:42:18.931 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:18.944+0000 7f3ee1ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:42:18.931 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:18.944+0000 7f3ee1ffb640 10 client.?.objecter ms_handle_connect 0x7f3ee41cc710 2024-03-29T10:42:18.931 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:18.944+0000 7f3ee1ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:42:18.931 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:18.944+0000 7f3ee1ffb640 1 -- 172.21.15.88:0/1965152841 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 440727415 0 0) 0x7f3ee41e1430 con 0x7f3ee40b0500 2024-03-29T10:42:18.932 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:18.944+0000 7f3ee1ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:42:18.932 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:18.944+0000 7f3ee1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:42:18.932 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:18.944+0000 7f3ee1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:42:18.932 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:18.944+0000 7f3ee1ffb640 10 monclient(hunting): my global_id is 4279 2024-03-29T10:42:18.932 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:18.944+0000 7f3ee1ffb640 1 -- 172.21.15.88:0/1965152841 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3ec8004eb0 con 0x7f3ee40b0500 2024-03-29T10:42:18.932 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:18.945+0000 7f3ee1ffb640 1 -- 172.21.15.88:0/1965152841 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3289696660 0 0) 0x7f3ee41e15c0 con 0x7f3ee41cc710 2024-03-29T10:42:18.932 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:18.945+0000 7f3ee1ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:42:18.932 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:18.945+0000 7f3ee1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:42:18.932 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:18.945+0000 7f3ee1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:42:18.932 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:18.945+0000 7f3ee1ffb640 10 monclient(hunting): my global_id is 4292 2024-03-29T10:42:18.932 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:18.945+0000 7f3ee1ffb640 1 -- 172.21.15.88:0/1965152841 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3ee41e1430 con 0x7f3ee41cc710 2024-03-29T10:42:18.937 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:18.946+0000 7f3ee1ffb640 1 -- 172.21.15.88:0/1965152841 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 357595858 0 0) 0x7f3ee41e1430 con 0x7f3ee41cc710 2024-03-29T10:42:18.937 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:18.946+0000 7f3ee1ffb640 1 -- 172.21.15.88:0/1965152841 --> v1:172.21.15.88:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3ec8005490 con 0x7f3ee41cc710 2024-03-29T10:42:18.937 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:18.946+0000 7f3ee1ffb640 1 -- 172.21.15.88:0/1965152841 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f3edc001510 con 0x7f3ee41cc710 2024-03-29T10:42:18.937 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:18.946+0000 7f3ee1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:18.937 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:18.946+0000 7f3ee1ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:42:18.937 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:18.946+0000 7f3ee1ffb640 10 monclient(hunting): dump: 2024-03-29T10:42:18.937 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:42:18.937 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:42:18.938 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:42:18.938 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:42:18.938 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:18.938 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:42:18.938 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:42:18.938 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:42:18.938 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:42:18.938 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:42:18.938 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:18.946+0000 7f3ee1ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:42:18.938 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:18.947+0000 7f3ee1ffb640 1 -- 172.21.15.88:0/1965152841 <== mon.2 v1:172.21.15.88:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3985991986 0 0) 0x7f3ec8005490 con 0x7f3ee41cc710 2024-03-29T10:42:18.938 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:18.947+0000 7f3ee1ffb640 1 -- 172.21.15.88:0/1965152841 >> v1:172.21.15.196:6789/0 conn(0x7f3ee40b0500 legacy=0x7f3ee41e0d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:18.938 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:18.947+0000 7f3ee1ffb640 1 -- 172.21.15.88:0/1965152841 >> v1:172.21.15.88:6789/0 conn(0x7f3ee40aeca0 legacy=0x7f3ee406fe20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:18.939 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:18.947+0000 7f3ee1ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:42:18.939 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:18.947+0000 7f3ee1ffb640 1 monclient: found mon.c 2024-03-29T10:42:18.939 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:18.947+0000 7f3ee1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:42:18.939 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:18.947+0000 7f3ee1ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-03-29T10:42:18.939 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:18.947+0000 7f3ee1ffb640 1 -- 172.21.15.88:0/1965152841 --> v1:172.21.15.88:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ee41ccba0 con 0x7f3ee41cc710 2024-03-29T10:42:18.939 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:18.947+0000 7f3ee1ffb640 10 monclient: _finish_auth 0 2024-03-29T10:42:18.939 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:18.947+0000 7f3ee1ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:42:18.939 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:18.947+0000 7f3ee1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:42:18.939 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:18.947+0000 7f3eeb84e640 5 monclient: authenticate success, global_id 4292 2024-03-29T10:42:18.939 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:18.947+0000 7f3eeb84e640 10 monclient: _renew_subs 2024-03-29T10:42:18.939 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:18.948+0000 7f3eeb84e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-03-29T10:42:18.939 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:18.948+0000 7f3eeb84e640 1 -- 172.21.15.88:0/1965152841 --> v1:172.21.15.88:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ee41ccdf0 con 0x7f3ee41cc710 2024-03-29T10:42:18.939 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:18.948+0000 7f3eeb84e640 10 client.4292.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:42:18.939 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:18.948+0000 7f3eeb84e640 10 monclient: _renew_subs 2024-03-29T10:42:18.940 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:18.948+0000 7f3eeb84e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-03-29T10:42:18.940 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:18.948+0000 7f3eeb84e640 1 -- 172.21.15.88:0/1965152841 --> v1:172.21.15.88:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ee41e44b0 con 0x7f3ee41cc710 2024-03-29T10:42:18.940 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:18.948+0000 7f3eeb84e640 1 librados: init done 2024-03-29T10:42:18.940 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:18.951+0000 7f3ee1ffb640 1 -- 172.21.15.88:0/1965152841 <== mon.2 v1:172.21.15.88:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3edc001a00 con 0x7f3ee41cc710 2024-03-29T10:42:18.940 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:18.951+0000 7f3ee1ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:42:18.940 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:18.951+0000 7f3ee1ffb640 1 -- 172.21.15.88:0/1965152841 <== mon.2 v1:172.21.15.88:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f3edc001510 con 0x7f3ee41cc710 2024-03-29T10:42:18.940 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:18.951+0000 7f3ee1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:18.940 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:18.951+0000 7f3ee1ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:42:18.940 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:18.951+0000 7f3ee1ffb640 10 monclient: dump: 2024-03-29T10:42:18.940 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:42:18.940 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:42:18.940 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:42:18.940 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:42:18.940 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:18.941 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:42:18.941 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:42:18.941 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:42:18.941 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:42:18.941 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:42:18.941 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:18.951+0000 7f3ee37fe640 4 set_mon_vals no callback set 2024-03-29T10:42:18.941 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:18.951+0000 7f3eeb84e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:42:18.941 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:18.952+0000 7f3ee1ffb640 1 -- 172.21.15.88:0/1965152841 <== mon.2 v1:172.21.15.88:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4235629220 0 0) 0x7f3edc019700 con 0x7f3ee41cc710 2024-03-29T10:42:18.941 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:18.952+0000 7f3eeb84e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:42:18.941 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:18.952+0000 7f3eeb84e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-03-29T10:42:18.941 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:18.952+0000 7f3eeb84e640 1 -- 172.21.15.88:0/1965152841 --> v1:172.21.15.88:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ec4005080 con 0x7f3ee41cc710 2024-03-29T10:42:18.941 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:18.952+0000 7f3ee1ffb640 1 -- 172.21.15.88:0/1965152841 <== mon.2 v1:172.21.15.88:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4257673582 0 0) 0x7f3edc0833b0 con 0x7f3ee41cc710 2024-03-29T10:42:18.941 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:18.952+0000 7f3ee1ffb640 10 client.4292.objecter ms_dispatch 0x7f3ee4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:42:18.941 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:18.952+0000 7f3ee1ffb640 3 client.4292.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:42:18.942 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:18.952+0000 7f3ee1ffb640 3 client.4292.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:42:18.942 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:18.952+0000 7f3ee1ffb640 20 client.4292.objecter dump_active .. 0 homeless 2024-03-29T10:42:18.942 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:18.955+0000 7f3ee95ea640 10 monclient: get_auth_request con 0x7f3ec806e220 auth_method 0 2024-03-29T10:42:18.942 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:18.955+0000 7f3ee1ffb640 10 client.4292.objecter ms_handle_connect 0x7f3ec806e220 2024-03-29T10:42:18.945 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:18.961+0000 7f3ee1ffb640 1 -- 172.21.15.88:0/1965152841 <== mon.2 v1:172.21.15.88:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f3edc04ec90 con 0x7f3ee41cc710 2024-03-29T10:42:18.945 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:18.961+0000 7f3ee1ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:42:18.945 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:18.961+0000 7f3ee1ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:42:19.285 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:19.301+0000 7f3eeb84e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:42:19.285 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:19.301+0000 7f3eeb84e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:42:19.285 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:19.301+0000 7f3eeb84e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-03-29T10:42:19.285 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:19.301+0000 7f3eeb84e640 1 -- 172.21.15.88:0/1965152841 --> v1:172.21.15.88:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3ec4005330 con 0x7f3ee41cc710 2024-03-29T10:42:19.286 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:19.302+0000 7f3ee1ffb640 1 -- 172.21.15.88:0/1965152841 <== mon.2 v1:172.21.15.88:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 344506374) 0x7f3edc050bc0 con 0x7f3ee41cc710 2024-03-29T10:42:19.286 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:19.302+0000 7f3ee1ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:42:19.286 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:19.302+0000 7f3ee1ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:42:19.286 INFO:teuthology.misc.health.smithi088.stdout: 2024-03-29T10:42:19.286 INFO:teuthology.misc.health.smithi088.stdout:{"epoch":10,"fsid":"869f5188-f92b-42dd-a46d-8c646e0388a9","created":"2024-03-29T10:40:40.285087+0000","modified":"2024-03-29T10:40:42.536230+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:40:42.536230+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a26c0c1c-8de0-438c-a3f6-3df10ea61574","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5de24915-8dec-49b6-8b16-f3170197c591","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ac054af3-d589-4565-9644-e6af75975c53","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3ec4f045-c714-4a22-bd1a-e07330b45ab3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"cddb9867-8e30-4951-9b13-3d1e78b9c45a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dc6d9911-785d-432f-8b53-56a1ccab302f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"77c88437-b587-496e-8b4d-1824ea8b5305","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ce963d96-5a8e-44c3-99de-c0dbebe234ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:42:19.289 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:19.306+0000 7f3eeb84e640 10 librados: watch_flush enter 2024-03-29T10:42:19.289 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:19.306+0000 7f3eeb84e640 10 librados: watch_flush exit 2024-03-29T10:42:19.289 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:19.306+0000 7f3eeb84e640 20 client.4292.objecter shutdown clearing up homeless session... 2024-03-29T10:42:19.289 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:19.306+0000 7f3eeb84e640 10 client.4292.objecter successfully canceled tick 2024-03-29T10:42:19.292 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:19.309+0000 7f3eeb84e640 1 -- 172.21.15.88:0/1965152841 >> v1:172.21.15.88:6816/1752861349 conn(0x7f3ec806e220 legacy=0x7f3ec80706e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:19.292 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:19.309+0000 7f3eeb84e640 10 monclient: shutdown 2024-03-29T10:42:19.293 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:19.309+0000 7f3eeb84e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:42:19.293 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:19.309+0000 7f3eeb84e640 1 -- 172.21.15.88:0/1965152841 >> v1:172.21.15.88:6790/0 conn(0x7f3ee41cc710 legacy=0x7f3ee41e3bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:19.293 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:19.309+0000 7f3eeb84e640 1 -- 172.21.15.88:0/1965152841 shutdown_connections 2024-03-29T10:42:19.293 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:19.309+0000 7f3eeb84e640 1 -- 172.21.15.88:0/1965152841 >> 172.21.15.88:0/1965152841 conn(0x7f3ee406b660 msgr2=0x7f3ee40a9c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:42:19.293 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:19.310+0000 7f3eeb84e640 1 -- 172.21.15.88:0/1965152841 shutdown_connections 2024-03-29T10:42:19.293 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:19.310+0000 7f3eeb84e640 1 -- 172.21.15.88:0/1965152841 wait complete. 2024-03-29T10:42:19.294 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:19.310+0000 7f3eeb84e640 1 librados: shutdown 2024-03-29T10:42:19.311 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:42:26.119 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:42:26.272 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:26.288+0000 7f001f59e640 10 monclient: get_monmap_and_config 2024-03-29T10:42:26.272 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:26.288+0000 7f001f59e640 10 monclient: build_initial_monmap 2024-03-29T10:42:26.272 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:26.288+0000 7f001f59e640 10 monclient: monmap: 2024-03-29T10:42:26.272 INFO:teuthology.misc.health.smithi088.stderr:epoch 0 2024-03-29T10:42:26.272 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:42:26.272 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:42:26.289114+0000 2024-03-29T10:42:26.272 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:42:26.289114+0000 2024-03-29T10:42:26.272 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 0 (unknown) 2024-03-29T10:42:26.272 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:42:26.272 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-03-29T10:42:26.272 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-29T10:42:26.273 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-b 2024-03-29T10:42:26.273 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:42:26.274 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:26.290+0000 7f001f59e640 1 Processor -- start 2024-03-29T10:42:26.274 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:26.291+0000 7f001f59e640 1 -- start start 2024-03-29T10:42:26.275 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:26.291+0000 7f001f59e640 10 monclient: init 2024-03-29T10:42:26.278 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:26.292+0000 7f001f59e640 10 monclient: _reopen_session rank -1 2024-03-29T10:42:26.279 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:26.292+0000 7f001f59e640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:42:26.279 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:26.292+0000 7f001f59e640 10 monclient(hunting): picked mon.noname-b con 0x7f0020070450 addr v1:172.21.15.88:6790/0 2024-03-29T10:42:26.279 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:26.292+0000 7f001f59e640 10 monclient(hunting): picked mon.noname-c con 0x7f00200b03b0 addr v1:172.21.15.196:6789/0 2024-03-29T10:42:26.279 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:26.292+0000 7f001f59e640 10 monclient(hunting): picked mon.noname-a con 0x7f002013d370 addr v1:172.21.15.88:6789/0 2024-03-29T10:42:26.279 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:26.292+0000 7f001f59e640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f002013fdc0 con 0x7f002013d370 2024-03-29T10:42:26.279 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:26.292+0000 7f001f59e640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0020071e40 con 0x7f00200b03b0 2024-03-29T10:42:26.279 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:26.292+0000 7f001f59e640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0020072120 con 0x7f0020070450 2024-03-29T10:42:26.279 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:26.292+0000 7f001f59e640 10 monclient(hunting): _renew_subs 2024-03-29T10:42:26.280 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:26.292+0000 7f001f59e640 10 monclient(hunting): authenticate will time out at 868.528157s 2024-03-29T10:42:26.282 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:26.298+0000 7f001dd9b640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f00200b03b0 0x7f002013cc60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.88:33233/0 (socket says 172.21.15.88:33233) 2024-03-29T10:42:26.282 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:26.298+0000 7f001dd9b640 1 -- 172.21.15.88:0/248432574 learned_addr learned my addr 172.21.15.88:0/248432574 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-29T10:42:26.283 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:26.300+0000 7f001d59a640 1 -- 172.21.15.88:0/248432574 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3322270509 0 0) 0x7f0020071e40 con 0x7f00200b03b0 2024-03-29T10:42:26.283 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:26.300+0000 7f001d59a640 10 monclient(hunting): _negotiate 2024-03-29T10:42:26.283 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:26.300+0000 7f001d59a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:42:26.283 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:26.300+0000 7f001d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:42:26.283 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:26.300+0000 7f001d59a640 10 monclient(hunting): my global_id is 4282 2024-03-29T10:42:26.283 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:26.300+0000 7f001d59a640 1 -- 172.21.15.88:0/248432574 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0008003520 con 0x7f00200b03b0 2024-03-29T10:42:26.283 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:26.300+0000 7f001d59a640 1 -- 172.21.15.88:0/248432574 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3358835722 0 0) 0x7f0020072120 con 0x7f0020070450 2024-03-29T10:42:26.283 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:26.300+0000 7f001d59a640 10 monclient(hunting): _negotiate 2024-03-29T10:42:26.283 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:26.300+0000 7f001d59a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:42:26.284 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:26.300+0000 7f001d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:42:26.284 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:26.300+0000 7f001d59a640 10 monclient(hunting): my global_id is 4295 2024-03-29T10:42:26.284 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:26.300+0000 7f001d59a640 1 -- 172.21.15.88:0/248432574 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f00080048a0 con 0x7f0020070450 2024-03-29T10:42:26.284 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:26.300+0000 7f001d59a640 1 -- 172.21.15.88:0/248432574 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2039132819 0 0) 0x7f00080048a0 con 0x7f0020070450 2024-03-29T10:42:26.284 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:26.300+0000 7f001d59a640 1 -- 172.21.15.88:0/248432574 >> v1:172.21.15.196:6789/0 conn(0x7f00200b03b0 legacy=0x7f002013cc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:26.285 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:26.300+0000 7f001d59a640 1 -- 172.21.15.88:0/248432574 >> v1:172.21.15.88:6789/0 conn(0x7f002013d370 legacy=0x7f002014b520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:26.285 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:26.300+0000 7f001d59a640 10 monclient: _finish_hunting 0 2024-03-29T10:42:26.285 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:26.300+0000 7f001d59a640 1 monclient: found mon.noname-b 2024-03-29T10:42:26.285 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:26.300+0000 7f001d59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:42:26.285 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:26.300+0000 7f001d59a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.88:6790/0 2024-03-29T10:42:26.285 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:26.300+0000 7f001d59a640 1 -- 172.21.15.88:0/248432574 --> v1:172.21.15.88:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0020140960 con 0x7f0020070450 2024-03-29T10:42:26.285 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:26.300+0000 7f001d59a640 10 monclient: _finish_auth 0 2024-03-29T10:42:26.285 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:26.300+0000 7f001d59a640 10 monclient: _check_auth_tickets 2024-03-29T10:42:26.285 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:26.300+0000 7f001d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:42:26.285 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:26.300+0000 7f001f59e640 5 monclient: authenticate success, global_id 4295 2024-03-29T10:42:26.285 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:26.300+0000 7f001f59e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:42:26.285 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:26.300+0000 7f001d59a640 1 -- 172.21.15.88:0/248432574 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f00180019b0 con 0x7f0020070450 2024-03-29T10:42:26.285 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:26.300+0000 7f001d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:26.285 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:26.300+0000 7f001d59a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:42:26.286 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:26.300+0000 7f001d59a640 10 monclient: dump: 2024-03-29T10:42:26.286 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:42:26.286 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:42:26.286 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:42:26.286 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:42:26.286 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:26.286 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:42:26.286 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:42:26.286 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:42:26.286 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:42:26.286 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:42:26.286 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:26.300+0000 7f001f59e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:42:26.286 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:26.300+0000 7f001d59a640 1 -- 172.21.15.88:0/248432574 <== mon.2 v1:172.21.15.88:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0018001c00 con 0x7f0020070450 2024-03-29T10:42:26.287 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:26.300+0000 7f001d59a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:42:26.287 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:26.300+0000 7f001d59a640 1 -- 172.21.15.88:0/248432574 <== mon.2 v1:172.21.15.88:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f00180019b0 con 0x7f0020070450 2024-03-29T10:42:26.287 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:26.300+0000 7f001d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:26.287 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:26.300+0000 7f001d59a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:42:26.287 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:26.300+0000 7f001d59a640 10 monclient: dump: 2024-03-29T10:42:26.287 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:42:26.287 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:42:26.287 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:42:26.287 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:42:26.287 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:26.287 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:42:26.287 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:42:26.287 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:42:26.287 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:42:26.288 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:42:26.288 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:26.301+0000 7f001f59e640 10 monclient: get_monmap_and_config success 2024-03-29T10:42:26.288 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:26.301+0000 7f001f59e640 4 set_mon_vals no callback set 2024-03-29T10:42:26.288 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:26.301+0000 7f001f59e640 10 monclient: shutdown 2024-03-29T10:42:26.288 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:26.302+0000 7f001f59e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:42:26.289 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:26.302+0000 7f001f59e640 1 -- 172.21.15.88:0/248432574 >> v1:172.21.15.88:6790/0 conn(0x7f0020070450 legacy=0x7f0020140250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:26.289 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:26.305+0000 7f001f59e640 1 -- 172.21.15.88:0/248432574 shutdown_connections 2024-03-29T10:42:26.291 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:26.305+0000 7f001f59e640 1 -- 172.21.15.88:0/248432574 >> 172.21.15.88:0/248432574 conn(0x7f00200671c0 msgr2=0x7f00200b59b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:42:26.291 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:26.305+0000 7f001f59e640 1 -- 172.21.15.88:0/248432574 shutdown_connections 2024-03-29T10:42:26.291 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:26.305+0000 7f001f59e640 1 -- 172.21.15.88:0/248432574 wait complete. 2024-03-29T10:42:26.295 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:26.309+0000 7f001f59e640 10 monclient: build_initial_monmap 2024-03-29T10:42:26.295 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:26.309+0000 7f001f59e640 10 monclient: monmap: 2024-03-29T10:42:26.295 INFO:teuthology.misc.health.smithi088.stderr:epoch 0 2024-03-29T10:42:26.295 INFO:teuthology.misc.health.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:42:26.295 INFO:teuthology.misc.health.smithi088.stderr:last_changed 0.000000 2024-03-29T10:42:26.295 INFO:teuthology.misc.health.smithi088.stderr:created 0.000000 2024-03-29T10:42:26.295 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 0 (unknown) 2024-03-29T10:42:26.295 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:42:26.296 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-03-29T10:42:26.296 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-29T10:42:26.296 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-c 2024-03-29T10:42:26.296 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:42:26.296 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:26.309+0000 7f001f59e640 1 librados: starting msgr at 2024-03-29T10:42:26.296 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:26.309+0000 7f001f59e640 1 librados: starting objecter 2024-03-29T10:42:26.296 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:26.309+0000 7f001f59e640 1 Processor -- start 2024-03-29T10:42:26.296 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:26.310+0000 7f001f59e640 1 -- start start 2024-03-29T10:42:26.296 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:26.310+0000 7f001f59e640 1 librados: setting wanted keys 2024-03-29T10:42:26.296 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:26.310+0000 7f001f59e640 1 librados: calling monclient init 2024-03-29T10:42:26.296 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:26.310+0000 7f001f59e640 10 monclient: init 2024-03-29T10:42:26.296 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:26.310+0000 7f001f59e640 10 monclient: _reopen_session rank -1 2024-03-29T10:42:26.296 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:26.310+0000 7f001f59e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:42:26.296 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:26.310+0000 7f001f59e640 10 monclient(hunting): picked mon.noname-b con 0x7f0020070450 addr v1:172.21.15.196:6789/0 2024-03-29T10:42:26.296 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:26.310+0000 7f001f59e640 10 monclient(hunting): picked mon.noname-c con 0x7f00200b03b0 addr v1:172.21.15.88:6790/0 2024-03-29T10:42:26.297 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:26.310+0000 7f001f59e640 10 monclient(hunting): picked mon.noname-a con 0x7f002013d370 addr v1:172.21.15.88:6789/0 2024-03-29T10:42:26.297 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:26.310+0000 7f001f59e640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00201425d0 con 0x7f002013d370 2024-03-29T10:42:26.297 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:26.310+0000 7f001f59e640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0020143870 con 0x7f0020070450 2024-03-29T10:42:26.297 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:26.310+0000 7f001f59e640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00201422a0 con 0x7f00200b03b0 2024-03-29T10:42:26.297 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:26.310+0000 7f001f59e640 10 monclient(hunting): _renew_subs 2024-03-29T10:42:26.297 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:26.310+0000 7f001f59e640 10 monclient(hunting): authenticate will time out at 868.546666s 2024-03-29T10:42:26.303 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:26.320+0000 7f001e59c640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f0020070450 0x7f00201416d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.88:33369/0 (socket says 172.21.15.88:33369) 2024-03-29T10:42:26.303 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:26.320+0000 7f001ed9d640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f002013d370 0x7f0020200600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:46299/0 (socket says 172.21.15.88:46299) 2024-03-29T10:42:26.303 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:26.320+0000 7f001ed9d640 1 -- 172.21.15.88:0/1738072108 learned_addr learned my addr 172.21.15.88:0/1738072108 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-29T10:42:26.304 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:26.321+0000 7f000effd640 10 client.?.objecter ms_handle_connect 0x7f002013d370 2024-03-29T10:42:26.304 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:26.321+0000 7f000effd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:42:26.305 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:26.321+0000 7f000effd640 1 -- 172.21.15.88:0/1738072108 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1376317908 0 0) 0x7f00201425d0 con 0x7f002013d370 2024-03-29T10:42:26.305 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:26.321+0000 7f000effd640 10 monclient(hunting): _negotiate 2024-03-29T10:42:26.305 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:26.321+0000 7f000effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:42:26.305 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:26.321+0000 7f000effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:42:26.305 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:26.321+0000 7f000effd640 10 monclient(hunting): my global_id is 4296 2024-03-29T10:42:26.305 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:26.321+0000 7f000effd640 1 -- 172.21.15.88:0/1738072108 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0000004eb0 con 0x7f002013d370 2024-03-29T10:42:26.306 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:26.322+0000 7f001ed9d640 1 -- 172.21.15.88:0/1738072108 >> v1:172.21.15.88:6789/0 conn(0x7f002013d370 legacy=0x7f0020200600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-03-29T10:42:26.306 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:26.322+0000 7f001ed9d640 1 -- 172.21.15.88:0/1738072108 >> v1:172.21.15.88:6789/0 conn(0x7f002013d370 legacy=0x7f0020200600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-29T10:42:26.306 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:26.322+0000 7f001ed9d640 1 --1- 172.21.15.88:0/1738072108 >> v1:172.21.15.88:6789/0 conn(0x7f002013d370 0x7f0020200600 :-1 s=OPENED pgs=182482 cs=1 l=1).handle_message read tag failed 2024-03-29T10:42:26.306 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:26.323+0000 7f001ed9d640 1 --1- 172.21.15.88:0/1738072108 >> v1:172.21.15.88:6789/0 conn(0x7f002013d370 0x7f0020200600 :-1 s=OPENED pgs=182482 cs=1 l=1).fault on lossy channel, failing 2024-03-29T10:42:26.308 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:26.325+0000 7f000effd640 10 client.?.objecter ms_handle_connect 0x7f00200b03b0 2024-03-29T10:42:26.308 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:26.325+0000 7f000effd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:42:26.308 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:26.325+0000 7f000effd640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.88:6789/0 2024-03-29T10:42:26.308 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:26.325+0000 7f000effd640 10 client.?.objecter ms_handle_connect 0x7f0020070450 2024-03-29T10:42:26.308 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:26.325+0000 7f000effd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:42:26.308 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:26.325+0000 7f000effd640 1 -- 172.21.15.88:0/1738072108 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 993195860 0 0) 0x7f00201422a0 con 0x7f00200b03b0 2024-03-29T10:42:26.308 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:26.325+0000 7f000effd640 10 monclient(hunting): _negotiate 2024-03-29T10:42:26.308 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:26.325+0000 7f000effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:42:26.308 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:26.325+0000 7f000effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:42:26.309 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:26.325+0000 7f000effd640 10 monclient(hunting): my global_id is 4298 2024-03-29T10:42:26.309 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:26.325+0000 7f000effd640 1 -- 172.21.15.88:0/1738072108 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f00201425d0 con 0x7f00200b03b0 2024-03-29T10:42:26.309 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:26.325+0000 7f000effd640 1 -- 172.21.15.88:0/1738072108 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1092804230 0 0) 0x7f0020143870 con 0x7f0020070450 2024-03-29T10:42:26.309 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:26.325+0000 7f000effd640 10 monclient(hunting): _negotiate 2024-03-29T10:42:26.309 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:26.325+0000 7f000effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:42:26.309 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:26.325+0000 7f000effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:42:26.310 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:26.325+0000 7f000effd640 10 monclient(hunting): my global_id is 4285 2024-03-29T10:42:26.310 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:26.325+0000 7f000effd640 1 -- 172.21.15.88:0/1738072108 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f00201422a0 con 0x7f0020070450 2024-03-29T10:42:26.310 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:26.325+0000 7f000effd640 1 -- 172.21.15.88:0/1738072108 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3265357972 0 0) 0x7f00201422a0 con 0x7f0020070450 2024-03-29T10:42:26.310 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:26.325+0000 7f000effd640 1 -- 172.21.15.88:0/1738072108 --> v1:172.21.15.196:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0020143870 con 0x7f0020070450 2024-03-29T10:42:26.310 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:26.325+0000 7f000effd640 1 -- 172.21.15.88:0/1738072108 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f00180020f0 con 0x7f0020070450 2024-03-29T10:42:26.310 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:26.325+0000 7f000effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:26.310 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:26.325+0000 7f000effd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:42:26.310 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:26.325+0000 7f000effd640 10 monclient(hunting): dump: 2024-03-29T10:42:26.310 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:42:26.310 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:42:26.310 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:42:26.310 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:42:26.310 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:26.310 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:42:26.311 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:42:26.311 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:42:26.311 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:42:26.311 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:42:26.311 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:26.325+0000 7f000effd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:42:26.311 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:26.326+0000 7f000effd640 1 -- 172.21.15.88:0/1738072108 <== mon.1 v1:172.21.15.196:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 296894911 0 0) 0x7f0020143870 con 0x7f0020070450 2024-03-29T10:42:26.311 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:26.326+0000 7f000effd640 1 -- 172.21.15.88:0/1738072108 >> v1:172.21.15.88:6790/0 conn(0x7f00200b03b0 legacy=0x7f0020137e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:26.311 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:26.326+0000 7f000effd640 1 -- 172.21.15.88:0/1738072108 >> v1:172.21.15.88:6789/0 conn(0x7f002013d370 legacy=0x7f0020200600 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-29T10:42:26.311 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:26.326+0000 7f000effd640 10 monclient: _finish_hunting 0 2024-03-29T10:42:26.311 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:26.326+0000 7f000effd640 1 monclient: found mon.b 2024-03-29T10:42:26.311 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:26.326+0000 7f000effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:42:26.311 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:26.326+0000 7f000effd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T10:42:26.311 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:26.326+0000 7f000effd640 1 -- 172.21.15.88:0/1738072108 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0020143ac0 con 0x7f0020070450 2024-03-29T10:42:26.311 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:26.326+0000 7f000effd640 10 monclient: _finish_auth 0 2024-03-29T10:42:26.312 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:26.326+0000 7f000effd640 10 monclient: _check_auth_tickets 2024-03-29T10:42:26.312 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:26.326+0000 7f000effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:42:26.312 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:26.326+0000 7f000effd640 1 -- 172.21.15.88:0/1738072108 <== mon.1 v1:172.21.15.196:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0018003c20 con 0x7f0020070450 2024-03-29T10:42:26.312 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:26.326+0000 7f000effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:42:26.313 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:26.326+0000 7f000ffff640 4 set_mon_vals no callback set 2024-03-29T10:42:26.313 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:26.327+0000 7f000effd640 1 -- 172.21.15.88:0/1738072108 <== mon.1 v1:172.21.15.196:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f0018003fb0 con 0x7f0020070450 2024-03-29T10:42:26.313 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:26.327+0000 7f000effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:26.313 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:26.327+0000 7f000effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:42:26.313 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:26.327+0000 7f000effd640 10 monclient: dump: 2024-03-29T10:42:26.313 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:42:26.313 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:42:26.313 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:42:26.313 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:42:26.313 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:26.313 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:42:26.313 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:42:26.313 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:42:26.314 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:42:26.314 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:42:26.314 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:26.328+0000 7f001f59e640 5 monclient: authenticate success, global_id 4285 2024-03-29T10:42:26.314 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:26.328+0000 7f001f59e640 10 monclient: _renew_subs 2024-03-29T10:42:26.314 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:26.328+0000 7f001f59e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T10:42:26.314 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:26.328+0000 7f001f59e640 1 -- 172.21.15.88:0/1738072108 --> v1:172.21.15.196:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0020141ed0 con 0x7f0020070450 2024-03-29T10:42:26.314 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:26.328+0000 7f001f59e640 10 client.4285.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:42:26.314 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:26.328+0000 7f001f59e640 10 monclient: _renew_subs 2024-03-29T10:42:26.314 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:26.328+0000 7f001f59e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T10:42:26.314 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:26.328+0000 7f001f59e640 1 -- 172.21.15.88:0/1738072108 --> v1:172.21.15.196:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0020135b10 con 0x7f0020070450 2024-03-29T10:42:26.314 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:26.328+0000 7f001f59e640 1 librados: init done 2024-03-29T10:42:26.315 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:26.331+0000 7f001f59e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:42:26.315 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:26.331+0000 7f001f59e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:42:26.315 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:26.331+0000 7f001f59e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T10:42:26.315 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:26.331+0000 7f001f59e640 1 -- 172.21.15.88:0/1738072108 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00201420c0 con 0x7f0020070450 2024-03-29T10:42:26.315 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:26.332+0000 7f000effd640 1 -- 172.21.15.88:0/1738072108 <== mon.1 v1:172.21.15.196:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4235629220 0 0) 0x7f0018004110 con 0x7f0020070450 2024-03-29T10:42:26.320 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:26.333+0000 7f000effd640 1 -- 172.21.15.88:0/1738072108 <== mon.1 v1:172.21.15.196:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4257673582 0 0) 0x7f0018084da0 con 0x7f0020070450 2024-03-29T10:42:26.320 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:26.336+0000 7f001dd9b640 10 monclient: get_auth_request con 0x7f000006dda0 auth_method 0 2024-03-29T10:42:26.320 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:26.336+0000 7f000effd640 10 client.4285.objecter ms_dispatch 0x7f0020000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:42:26.320 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:26.336+0000 7f000effd640 3 client.4285.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:42:26.320 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:26.336+0000 7f000effd640 3 client.4285.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:42:26.320 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:26.336+0000 7f000effd640 20 client.4285.objecter dump_active .. 0 homeless 2024-03-29T10:42:26.320 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:26.337+0000 7f000effd640 10 client.4285.objecter ms_handle_connect 0x7f000006dda0 2024-03-29T10:42:26.324 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:26.341+0000 7f000effd640 1 -- 172.21.15.88:0/1738072108 <== mon.1 v1:172.21.15.196:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f00180504b0 con 0x7f0020070450 2024-03-29T10:42:26.324 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:26.341+0000 7f000effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:42:26.324 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:26.341+0000 7f000effd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:42:26.660 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:26.676+0000 7f001f59e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:42:26.660 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:26.676+0000 7f001f59e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:42:26.660 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:26.676+0000 7f001f59e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T10:42:26.660 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:26.676+0000 7f001f59e640 1 -- 172.21.15.88:0/1738072108 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f002004f730 con 0x7f0020070450 2024-03-29T10:42:26.661 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:26.677+0000 7f000effd640 1 -- 172.21.15.88:0/1738072108 <== mon.1 v1:172.21.15.196:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 344506374) 0x7f00180525b0 con 0x7f0020070450 2024-03-29T10:42:26.661 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:26.677+0000 7f000effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:42:26.661 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:26.677+0000 7f000effd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:42:26.661 INFO:teuthology.misc.health.smithi088.stdout: 2024-03-29T10:42:26.661 INFO:teuthology.misc.health.smithi088.stdout:{"epoch":10,"fsid":"869f5188-f92b-42dd-a46d-8c646e0388a9","created":"2024-03-29T10:40:40.285087+0000","modified":"2024-03-29T10:40:42.536230+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:40:42.536230+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a26c0c1c-8de0-438c-a3f6-3df10ea61574","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5de24915-8dec-49b6-8b16-f3170197c591","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ac054af3-d589-4565-9644-e6af75975c53","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3ec4f045-c714-4a22-bd1a-e07330b45ab3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"cddb9867-8e30-4951-9b13-3d1e78b9c45a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dc6d9911-785d-432f-8b53-56a1ccab302f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"77c88437-b587-496e-8b4d-1824ea8b5305","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ce963d96-5a8e-44c3-99de-c0dbebe234ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:42:26.665 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:26.681+0000 7f001f59e640 10 librados: watch_flush enter 2024-03-29T10:42:26.665 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:26.681+0000 7f001f59e640 10 librados: watch_flush exit 2024-03-29T10:42:26.665 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:26.681+0000 7f001f59e640 20 client.4285.objecter shutdown clearing up homeless session... 2024-03-29T10:42:26.665 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:26.681+0000 7f001f59e640 10 client.4285.objecter successfully canceled tick 2024-03-29T10:42:26.665 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:26.681+0000 7f001f59e640 1 -- 172.21.15.88:0/1738072108 >> v1:172.21.15.88:6816/1752861349 conn(0x7f000006dda0 legacy=0x7f0000070260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:26.665 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:26.681+0000 7f001f59e640 10 monclient: shutdown 2024-03-29T10:42:26.665 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:26.681+0000 7f001f59e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:42:26.665 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:26.681+0000 7f001f59e640 1 -- 172.21.15.88:0/1738072108 >> v1:172.21.15.196:6789/0 conn(0x7f0020070450 legacy=0x7f00201416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:26.666 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:26.683+0000 7f001f59e640 1 -- 172.21.15.88:0/1738072108 shutdown_connections 2024-03-29T10:42:26.666 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:26.683+0000 7f001f59e640 1 -- 172.21.15.88:0/1738072108 >> 172.21.15.88:0/1738072108 conn(0x7f00200671c0 msgr2=0x7f0020069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:42:26.667 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:26.684+0000 7f001f59e640 1 -- 172.21.15.88:0/1738072108 shutdown_connections 2024-03-29T10:42:26.667 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:26.684+0000 7f001f59e640 1 -- 172.21.15.88:0/1738072108 wait complete. 2024-03-29T10:42:26.667 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:26.684+0000 7f001f59e640 1 librados: shutdown 2024-03-29T10:42:26.684 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:42:33.492 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:42:33.652 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:33.668+0000 7fad4624a640 10 monclient: get_monmap_and_config 2024-03-29T10:42:33.652 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:33.668+0000 7fad4624a640 10 monclient: build_initial_monmap 2024-03-29T10:42:33.652 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:33.668+0000 7fad4624a640 10 monclient: monmap: 2024-03-29T10:42:33.652 INFO:teuthology.misc.health.smithi088.stderr:epoch 0 2024-03-29T10:42:33.652 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:42:33.653 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:42:33.669601+0000 2024-03-29T10:42:33.653 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:42:33.669601+0000 2024-03-29T10:42:33.653 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 0 (unknown) 2024-03-29T10:42:33.653 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:42:33.653 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-03-29T10:42:33.653 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-29T10:42:33.653 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-b 2024-03-29T10:42:33.653 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:42:33.655 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:33.671+0000 7fad4624a640 1 Processor -- start 2024-03-29T10:42:33.655 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:33.672+0000 7fad4624a640 1 -- start start 2024-03-29T10:42:33.655 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:33.672+0000 7fad4624a640 10 monclient: init 2024-03-29T10:42:33.657 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:33.672+0000 7fad4624a640 10 monclient: _reopen_session rank -1 2024-03-29T10:42:33.657 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:33.672+0000 7fad4624a640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:42:33.658 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:33.673+0000 7fad4624a640 10 monclient(hunting): picked mon.noname-a con 0x7fad400ae250 addr v1:172.21.15.88:6789/0 2024-03-29T10:42:33.658 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:33.673+0000 7fad4624a640 10 monclient(hunting): picked mon.noname-c con 0x7fad4013f970 addr v1:172.21.15.196:6789/0 2024-03-29T10:42:33.658 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:33.673+0000 7fad4624a640 10 monclient(hunting): picked mon.noname-b con 0x7fad40142800 addr v1:172.21.15.88:6790/0 2024-03-29T10:42:33.658 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:33.673+0000 7fad4624a640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad400afe00 con 0x7fad400ae250 2024-03-29T10:42:33.658 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:33.673+0000 7fad4624a640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad40142c10 con 0x7fad4013f970 2024-03-29T10:42:33.658 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:33.673+0000 7fad4624a640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad400afb20 con 0x7fad40142800 2024-03-29T10:42:33.658 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:33.673+0000 7fad4624a640 10 monclient(hunting): _renew_subs 2024-03-29T10:42:33.658 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:33.673+0000 7fad4624a640 10 monclient(hunting): authenticate will time out at 875.908974s 2024-03-29T10:42:33.660 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:33.677+0000 7fad3effd640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7fad4013f970 0x7fad401420f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.88:51303/0 (socket says 172.21.15.88:51303) 2024-03-29T10:42:33.660 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:33.677+0000 7fad3f7fe640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fad400ae250 0x7fad4013f260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:40141/0 (socket says 172.21.15.88:40141) 2024-03-29T10:42:33.661 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:33.677+0000 7fad3f7fe640 1 -- 172.21.15.88:0/1156269222 learned_addr learned my addr 172.21.15.88:0/1156269222 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-29T10:42:33.662 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:33.677+0000 7fad3e7fc640 1 -- 172.21.15.88:0/1156269222 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1438258402 0 0) 0x7fad400afe00 con 0x7fad400ae250 2024-03-29T10:42:33.662 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:33.677+0000 7fad3e7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:42:33.662 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:33.677+0000 7fad3e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:42:33.662 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:33.677+0000 7fad3e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:42:33.662 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:33.677+0000 7fad3e7fc640 10 monclient(hunting): my global_id is 4299 2024-03-29T10:42:33.662 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:33.677+0000 7fad3e7fc640 1 -- 172.21.15.88:0/1156269222 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fad28003520 con 0x7fad400ae250 2024-03-29T10:42:33.662 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:33.678+0000 7fad3e7fc640 1 -- 172.21.15.88:0/1156269222 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 670441566 0 0) 0x7fad400afb20 con 0x7fad40142800 2024-03-29T10:42:33.662 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:33.678+0000 7fad3e7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:42:33.662 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:33.678+0000 7fad3e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:42:33.662 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:33.678+0000 7fad3e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:42:33.662 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:33.678+0000 7fad3e7fc640 10 monclient(hunting): my global_id is 4301 2024-03-29T10:42:33.663 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:33.678+0000 7fad3e7fc640 1 -- 172.21.15.88:0/1156269222 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fad400afe00 con 0x7fad40142800 2024-03-29T10:42:33.663 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:33.678+0000 7fad3e7fc640 1 -- 172.21.15.88:0/1156269222 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3938683406 0 0) 0x7fad28003520 con 0x7fad400ae250 2024-03-29T10:42:33.663 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:33.678+0000 7fad3e7fc640 1 -- 172.21.15.88:0/1156269222 >> v1:172.21.15.88:6790/0 conn(0x7fad40142800 legacy=0x7fad4014da10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:33.663 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:33.678+0000 7fad3e7fc640 1 -- 172.21.15.88:0/1156269222 >> v1:172.21.15.196:6789/0 conn(0x7fad4013f970 legacy=0x7fad401420f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:33.663 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:33.678+0000 7fad3e7fc640 10 monclient: _finish_hunting 0 2024-03-29T10:42:33.663 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:33.678+0000 7fad3e7fc640 1 monclient: found mon.noname-a 2024-03-29T10:42:33.663 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:33.678+0000 7fad3e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:42:33.663 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:33.678+0000 7fad3e7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.88:6789/0 2024-03-29T10:42:33.663 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:33.678+0000 7fad3e7fc640 1 -- 172.21.15.88:0/1156269222 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad400aab80 con 0x7fad400ae250 2024-03-29T10:42:33.663 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:33.678+0000 7fad3e7fc640 10 monclient: _finish_auth 0 2024-03-29T10:42:33.663 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:33.678+0000 7fad3e7fc640 10 monclient: _check_auth_tickets 2024-03-29T10:42:33.663 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:33.678+0000 7fad3e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:42:33.663 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:33.678+0000 7fad3e7fc640 1 -- 172.21.15.88:0/1156269222 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7fad34001690 con 0x7fad400ae250 2024-03-29T10:42:33.664 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:33.678+0000 7fad3e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:33.664 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:33.678+0000 7fad3e7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:42:33.664 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:33.678+0000 7fad3e7fc640 10 monclient: dump: 2024-03-29T10:42:33.664 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:42:33.664 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:42:33.664 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:42:33.664 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:42:33.664 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:33.664 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:42:33.664 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:42:33.665 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:42:33.665 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:42:33.665 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:42:33.665 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:33.678+0000 7fad3e7fc640 1 -- 172.21.15.88:0/1156269222 <== mon.0 v1:172.21.15.88:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fad340017f0 con 0x7fad400ae250 2024-03-29T10:42:33.665 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:33.678+0000 7fad3e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:42:33.665 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:33.678+0000 7fad3e7fc640 1 -- 172.21.15.88:0/1156269222 <== mon.0 v1:172.21.15.88:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7fad34001c00 con 0x7fad400ae250 2024-03-29T10:42:33.665 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:33.678+0000 7fad3e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:33.665 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:33.678+0000 7fad3e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:42:33.665 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:33.678+0000 7fad3e7fc640 10 monclient: dump: 2024-03-29T10:42:33.665 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:42:33.665 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:42:33.665 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:42:33.666 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:42:33.666 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:33.666 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:42:33.666 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:42:33.666 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:42:33.666 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:42:33.666 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:42:33.666 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:33.678+0000 7fad4624a640 5 monclient: authenticate success, global_id 4299 2024-03-29T10:42:33.666 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:33.678+0000 7fad4624a640 10 monclient: get_monmap_and_config success 2024-03-29T10:42:33.666 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:33.678+0000 7fad4624a640 4 set_mon_vals no callback set 2024-03-29T10:42:33.668 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:33.680+0000 7fad4624a640 10 monclient: shutdown 2024-03-29T10:42:33.668 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:33.680+0000 7fad4624a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:42:33.668 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:33.680+0000 7fad4624a640 1 -- 172.21.15.88:0/1156269222 >> v1:172.21.15.88:6789/0 conn(0x7fad400ae250 legacy=0x7fad4013f260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:33.668 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:33.680+0000 7fad4624a640 1 -- 172.21.15.88:0/1156269222 shutdown_connections 2024-03-29T10:42:33.668 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:33.680+0000 7fad4624a640 1 -- 172.21.15.88:0/1156269222 >> 172.21.15.88:0/1156269222 conn(0x7fad400b35e0 msgr2=0x7fad400b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:42:33.668 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:33.681+0000 7fad4624a640 1 -- 172.21.15.88:0/1156269222 shutdown_connections 2024-03-29T10:42:33.668 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:33.681+0000 7fad4624a640 1 -- 172.21.15.88:0/1156269222 wait complete. 2024-03-29T10:42:33.669 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:33.682+0000 7fad4624a640 10 monclient: build_initial_monmap 2024-03-29T10:42:33.669 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:33.682+0000 7fad4624a640 10 monclient: monmap: 2024-03-29T10:42:33.669 INFO:teuthology.misc.health.smithi088.stderr:epoch 0 2024-03-29T10:42:33.669 INFO:teuthology.misc.health.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:42:33.669 INFO:teuthology.misc.health.smithi088.stderr:last_changed 0.000000 2024-03-29T10:42:33.669 INFO:teuthology.misc.health.smithi088.stderr:created 0.000000 2024-03-29T10:42:33.669 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 0 (unknown) 2024-03-29T10:42:33.669 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:42:33.669 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-03-29T10:42:33.669 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-29T10:42:33.669 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-c 2024-03-29T10:42:33.669 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:42:33.669 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:33.682+0000 7fad4624a640 1 librados: starting msgr at 2024-03-29T10:42:33.670 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:33.682+0000 7fad4624a640 1 librados: starting objecter 2024-03-29T10:42:33.670 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:33.682+0000 7fad4624a640 1 Processor -- start 2024-03-29T10:42:33.670 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:33.683+0000 7fad4624a640 1 -- start start 2024-03-29T10:42:33.670 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:33.683+0000 7fad4624a640 1 librados: setting wanted keys 2024-03-29T10:42:33.670 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:33.683+0000 7fad4624a640 1 librados: calling monclient init 2024-03-29T10:42:33.670 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:33.683+0000 7fad4624a640 10 monclient: init 2024-03-29T10:42:33.670 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:33.683+0000 7fad4624a640 10 monclient: _reopen_session rank -1 2024-03-29T10:42:33.670 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:33.684+0000 7fad4624a640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:42:33.670 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:33.684+0000 7fad4624a640 10 monclient(hunting): picked mon.noname-a con 0x7fad400ae250 addr v1:172.21.15.88:6789/0 2024-03-29T10:42:33.671 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:33.686+0000 7fad4624a640 10 monclient(hunting): picked mon.noname-c con 0x7fad4013f970 addr v1:172.21.15.88:6790/0 2024-03-29T10:42:33.671 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:33.686+0000 7fad3effd640 1 --1- >> v1:172.21.15.88:6790/0 conn(0x7fad4013f970 0x7fad401379e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6790/0 says I am v1:172.21.15.88:60163/0 (socket says 172.21.15.88:60163) 2024-03-29T10:42:33.671 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:33.686+0000 7fad3effd640 1 -- 172.21.15.88:0/2599832211 learned_addr learned my addr 172.21.15.88:0/2599832211 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-29T10:42:33.671 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:33.687+0000 7fad4624a640 10 monclient(hunting): picked mon.noname-b con 0x7fad40142800 addr v1:172.21.15.196:6789/0 2024-03-29T10:42:33.672 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:33.687+0000 7fad4624a640 1 -- 172.21.15.88:0/2599832211 --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad40138800 con 0x7fad400ae250 2024-03-29T10:42:33.672 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:33.687+0000 7fad4624a640 1 -- 172.21.15.88:0/2599832211 --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad40145dd0 con 0x7fad40142800 2024-03-29T10:42:33.672 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:33.687+0000 7fad4624a640 1 -- 172.21.15.88:0/2599832211 --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad401460b0 con 0x7fad4013f970 2024-03-29T10:42:33.672 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:33.687+0000 7fad4624a640 10 monclient(hunting): _renew_subs 2024-03-29T10:42:33.672 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:33.687+0000 7fad4624a640 10 monclient(hunting): authenticate will time out at 875.923159s 2024-03-29T10:42:33.673 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:33.689+0000 7fad27fff640 10 client.?.objecter ms_handle_connect 0x7fad4013f970 2024-03-29T10:42:33.673 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:33.689+0000 7fad27fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:42:33.673 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:33.689+0000 7fad27fff640 1 -- 172.21.15.88:0/2599832211 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3315783734 0 0) 0x7fad401460b0 con 0x7fad4013f970 2024-03-29T10:42:33.673 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:33.689+0000 7fad27fff640 10 monclient(hunting): _negotiate 2024-03-29T10:42:33.673 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:33.689+0000 7fad27fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:42:33.673 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:33.689+0000 7fad27fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:42:33.673 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:33.689+0000 7fad27fff640 10 monclient(hunting): my global_id is 4304 2024-03-29T10:42:33.673 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:33.689+0000 7fad27fff640 1 -- 172.21.15.88:0/2599832211 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fad2c004eb0 con 0x7fad4013f970 2024-03-29T10:42:33.675 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:33.691+0000 7fad27fff640 1 -- 172.21.15.88:0/2599832211 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3216991642 0 0) 0x7fad2c004eb0 con 0x7fad4013f970 2024-03-29T10:42:33.675 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:33.691+0000 7fad27fff640 1 -- 172.21.15.88:0/2599832211 --> v1:172.21.15.88:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fad401460b0 con 0x7fad4013f970 2024-03-29T10:42:33.675 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:33.691+0000 7fad27fff640 1 -- 172.21.15.88:0/2599832211 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7fad300019f0 con 0x7fad4013f970 2024-03-29T10:42:33.675 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:33.691+0000 7fad27fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:33.675 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:33.691+0000 7fad27fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:42:33.675 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:33.691+0000 7fad27fff640 10 monclient(hunting): dump: 2024-03-29T10:42:33.675 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:42:33.675 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:42:33.675 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:42:33.675 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:42:33.675 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:33.675 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:42:33.675 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:42:33.676 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:42:33.676 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:42:33.676 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:42:33.676 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:33.691+0000 7fad27fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:42:33.676 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:33.691+0000 7fad27fff640 1 -- 172.21.15.88:0/2599832211 <== mon.2 v1:172.21.15.88:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3663538682 0 0) 0x7fad401460b0 con 0x7fad4013f970 2024-03-29T10:42:33.676 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:33.691+0000 7fad27fff640 1 -- 172.21.15.88:0/2599832211 >> v1:172.21.15.196:6789/0 conn(0x7fad40142800 legacy=0x7fad401380f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:33.676 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:33.691+0000 7fad27fff640 1 -- 172.21.15.88:0/2599832211 >> v1:172.21.15.88:6789/0 conn(0x7fad400ae250 legacy=0x7fad4013a880 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:42:33.678 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:33.693+0000 7fad27fff640 10 monclient: _finish_hunting 0 2024-03-29T10:42:33.678 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:33.693+0000 7fad27fff640 1 monclient: found mon.c 2024-03-29T10:42:33.678 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:33.693+0000 7fad27fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:42:33.678 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:33.693+0000 7fad27fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-03-29T10:42:33.678 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:33.693+0000 7fad27fff640 1 -- 172.21.15.88:0/2599832211 --> v1:172.21.15.88:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad40146390 con 0x7fad4013f970 2024-03-29T10:42:33.679 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:33.693+0000 7fad27fff640 10 monclient: _finish_auth 0 2024-03-29T10:42:33.679 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:33.693+0000 7fad27fff640 10 monclient: _check_auth_tickets 2024-03-29T10:42:33.679 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:33.693+0000 7fad27fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:42:33.679 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:33.693+0000 7fad27fff640 1 -- 172.21.15.88:0/2599832211 <== mon.2 v1:172.21.15.88:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fad30001b50 con 0x7fad4013f970 2024-03-29T10:42:33.679 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:33.693+0000 7fad27fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:42:33.679 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:33.693+0000 7fad27fff640 1 -- 172.21.15.88:0/2599832211 <== mon.2 v1:172.21.15.88:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7fad30004c30 con 0x7fad4013f970 2024-03-29T10:42:33.679 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:33.693+0000 7fad27fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:33.679 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:33.693+0000 7fad27fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:42:33.679 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:33.693+0000 7fad27fff640 10 monclient: dump: 2024-03-29T10:42:33.679 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:42:33.679 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:42:33.679 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:42:33.679 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:42:33.679 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:33.680 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:42:33.680 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:42:33.680 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:42:33.680 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:42:33.680 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:42:33.680 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:33.694+0000 7fad4624a640 5 monclient: authenticate success, global_id 4304 2024-03-29T10:42:33.680 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:33.694+0000 7fad4624a640 10 monclient: _renew_subs 2024-03-29T10:42:33.680 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:33.694+0000 7fad4624a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-03-29T10:42:33.680 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:33.694+0000 7fad4624a640 1 -- 172.21.15.88:0/2599832211 --> v1:172.21.15.88:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad40144c70 con 0x7fad4013f970 2024-03-29T10:42:33.680 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:33.694+0000 7fad3dffb640 4 set_mon_vals no callback set 2024-03-29T10:42:33.680 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:33.694+0000 7fad4624a640 10 client.4304.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:42:33.680 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:33.694+0000 7fad4624a640 10 monclient: _renew_subs 2024-03-29T10:42:33.680 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:33.694+0000 7fad4624a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-03-29T10:42:33.680 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:33.694+0000 7fad4624a640 1 -- 172.21.15.88:0/2599832211 --> v1:172.21.15.88:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fad4013af90 con 0x7fad4013f970 2024-03-29T10:42:33.681 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:33.694+0000 7fad4624a640 1 librados: init done 2024-03-29T10:42:33.682 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:33.698+0000 7fad27fff640 1 -- 172.21.15.88:0/2599832211 <== mon.2 v1:172.21.15.88:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4235629220 0 0) 0x7fad30004d90 con 0x7fad4013f970 2024-03-29T10:42:33.682 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:33.698+0000 7fad27fff640 1 -- 172.21.15.88:0/2599832211 <== mon.2 v1:172.21.15.88:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4257673582 0 0) 0x7fad30084c10 con 0x7fad4013f970 2024-03-29T10:42:33.682 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:33.698+0000 7fad27fff640 10 client.4304.objecter ms_dispatch 0x7fad40000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:42:33.682 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:33.698+0000 7fad27fff640 3 client.4304.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:42:33.682 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:33.698+0000 7fad27fff640 3 client.4304.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:42:33.682 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:33.698+0000 7fad27fff640 20 client.4304.objecter dump_active .. 0 homeless 2024-03-29T10:42:33.682 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:33.699+0000 7fad3f7fe640 10 monclient: get_auth_request con 0x7fad2c06dc80 auth_method 0 2024-03-29T10:42:33.683 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:33.699+0000 7fad27fff640 10 client.4304.objecter ms_handle_connect 0x7fad2c06dc80 2024-03-29T10:42:33.686 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:33.703+0000 7fad4624a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:42:33.686 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:33.703+0000 7fad4624a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:42:33.686 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:33.703+0000 7fad4624a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-03-29T10:42:33.686 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:33.703+0000 7fad4624a640 1 -- 172.21.15.88:0/2599832211 --> v1:172.21.15.88:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad401432e0 con 0x7fad4013f970 2024-03-29T10:42:33.694 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:33.710+0000 7fad27fff640 1 -- 172.21.15.88:0/2599832211 <== mon.2 v1:172.21.15.88:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fad300504f0 con 0x7fad4013f970 2024-03-29T10:42:33.694 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:33.710+0000 7fad27fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:42:33.694 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:33.710+0000 7fad27fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:42:34.025 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:34.040+0000 7fad4624a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:42:34.025 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:34.040+0000 7fad4624a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:42:34.026 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:34.040+0000 7fad4624a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-03-29T10:42:34.027 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:34.040+0000 7fad4624a640 1 -- 172.21.15.88:0/2599832211 --> v1:172.21.15.88:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fad40075f10 con 0x7fad4013f970 2024-03-29T10:42:34.027 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:34.043+0000 7fad27fff640 1 -- 172.21.15.88:0/2599832211 <== mon.2 v1:172.21.15.88:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 344506374) 0x7fad30052420 con 0x7fad4013f970 2024-03-29T10:42:34.027 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:34.043+0000 7fad27fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:42:34.027 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:34.043+0000 7fad27fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:42:34.027 INFO:teuthology.misc.health.smithi088.stdout: 2024-03-29T10:42:34.027 INFO:teuthology.misc.health.smithi088.stdout:{"epoch":10,"fsid":"869f5188-f92b-42dd-a46d-8c646e0388a9","created":"2024-03-29T10:40:40.285087+0000","modified":"2024-03-29T10:40:42.536230+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:40:42.536230+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a26c0c1c-8de0-438c-a3f6-3df10ea61574","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5de24915-8dec-49b6-8b16-f3170197c591","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ac054af3-d589-4565-9644-e6af75975c53","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3ec4f045-c714-4a22-bd1a-e07330b45ab3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"cddb9867-8e30-4951-9b13-3d1e78b9c45a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dc6d9911-785d-432f-8b53-56a1ccab302f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"77c88437-b587-496e-8b4d-1824ea8b5305","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ce963d96-5a8e-44c3-99de-c0dbebe234ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:42:34.031 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:34.046+0000 7fad4624a640 10 librados: watch_flush enter 2024-03-29T10:42:34.031 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:34.046+0000 7fad4624a640 10 librados: watch_flush exit 2024-03-29T10:42:34.031 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:34.046+0000 7fad4624a640 20 client.4304.objecter shutdown clearing up homeless session... 2024-03-29T10:42:34.031 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:34.046+0000 7fad4624a640 10 client.4304.objecter successfully canceled tick 2024-03-29T10:42:34.031 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:34.046+0000 7fad4624a640 1 -- 172.21.15.88:0/2599832211 >> v1:172.21.15.88:6816/1752861349 conn(0x7fad2c06dc80 legacy=0x7fad2c070140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:34.031 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:34.046+0000 7fad4624a640 10 monclient: shutdown 2024-03-29T10:42:34.031 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:34.046+0000 7fad4624a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:42:34.032 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:34.046+0000 7fad4624a640 1 -- 172.21.15.88:0/2599832211 >> v1:172.21.15.88:6790/0 conn(0x7fad4013f970 legacy=0x7fad401379e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:34.032 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:34.049+0000 7fad4624a640 1 -- 172.21.15.88:0/2599832211 shutdown_connections 2024-03-29T10:42:34.032 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:34.049+0000 7fad4624a640 1 -- 172.21.15.88:0/2599832211 >> 172.21.15.88:0/2599832211 conn(0x7fad400b35e0 msgr2=0x7fad400b39c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:42:34.032 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:34.049+0000 7fad4624a640 1 -- 172.21.15.88:0/2599832211 shutdown_connections 2024-03-29T10:42:34.033 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:34.049+0000 7fad4624a640 1 -- 172.21.15.88:0/2599832211 wait complete. 2024-03-29T10:42:34.033 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:34.049+0000 7fad4624a640 1 librados: shutdown 2024-03-29T10:42:34.050 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:42:40.857 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:42:41.014 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:41.028+0000 7fc5cce06640 10 monclient: get_monmap_and_config 2024-03-29T10:42:41.015 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:41.028+0000 7fc5cce06640 10 monclient: build_initial_monmap 2024-03-29T10:42:41.015 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:41.028+0000 7fc5cce06640 10 monclient: monmap: 2024-03-29T10:42:41.015 INFO:teuthology.misc.health.smithi088.stderr:epoch 0 2024-03-29T10:42:41.015 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:42:41.015 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:42:41.029993+0000 2024-03-29T10:42:41.015 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:42:41.029993+0000 2024-03-29T10:42:41.015 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 0 (unknown) 2024-03-29T10:42:41.015 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:42:41.015 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-03-29T10:42:41.015 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-29T10:42:41.015 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-b 2024-03-29T10:42:41.015 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:42:41.015 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:41.029+0000 7fc5cce06640 1 Processor -- start 2024-03-29T10:42:41.015 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:41.029+0000 7fc5cce06640 1 -- start start 2024-03-29T10:42:41.016 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:41.029+0000 7fc5cce06640 10 monclient: init 2024-03-29T10:42:41.016 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:41.030+0000 7fc5cce06640 10 monclient: _reopen_session rank -1 2024-03-29T10:42:41.016 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:41.030+0000 7fc5cce06640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:42:41.016 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:41.030+0000 7fc5cce06640 10 monclient(hunting): picked mon.noname-c con 0x7fc5c80ad6c0 addr v1:172.21.15.196:6789/0 2024-03-29T10:42:41.016 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:41.030+0000 7fc5cce06640 10 monclient(hunting): picked mon.noname-a con 0x7fc5c80af6c0 addr v1:172.21.15.88:6789/0 2024-03-29T10:42:41.016 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:41.030+0000 7fc5cce06640 10 monclient(hunting): picked mon.noname-b con 0x7fc5c814fdc0 addr v1:172.21.15.88:6790/0 2024-03-29T10:42:41.016 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:41.030+0000 7fc5cce06640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5c80a94e0 con 0x7fc5c80af6c0 2024-03-29T10:42:41.016 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:41.030+0000 7fc5cce06640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5c80af3a0 con 0x7fc5c80ad6c0 2024-03-29T10:42:41.016 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:41.031+0000 7fc5cce06640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5c80afb60 con 0x7fc5c814fdc0 2024-03-29T10:42:41.016 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:41.031+0000 7fc5cce06640 10 monclient(hunting): _renew_subs 2024-03-29T10:42:41.016 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:41.031+0000 7fc5cce06640 10 monclient(hunting): authenticate will time out at 883.266902s 2024-03-29T10:42:41.023 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:41.034+0000 7fc5c659c640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7fc5c80ad6c0 0x7fc5c814f6b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.88:37407/0 (socket says 172.21.15.88:37407) 2024-03-29T10:42:41.023 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:41.034+0000 7fc5c659c640 1 -- 172.21.15.88:0/4168542110 learned_addr learned my addr 172.21.15.88:0/4168542110 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-29T10:42:41.023 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:41.036+0000 7fc5c559a640 1 -- 172.21.15.88:0/4168542110 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1568395486 0 0) 0x7fc5c80afb60 con 0x7fc5c814fdc0 2024-03-29T10:42:41.023 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:41.036+0000 7fc5c559a640 10 monclient(hunting): _negotiate 2024-03-29T10:42:41.024 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:41.036+0000 7fc5c559a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:42:41.024 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:41.036+0000 7fc5c559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:42:41.024 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:41.036+0000 7fc5c559a640 10 monclient(hunting): my global_id is 4307 2024-03-29T10:42:41.024 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:41.036+0000 7fc5c559a640 1 -- 172.21.15.88:0/4168542110 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc5b0003520 con 0x7fc5c814fdc0 2024-03-29T10:42:41.024 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:41.036+0000 7fc5c559a640 1 -- 172.21.15.88:0/4168542110 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2860069125 0 0) 0x7fc5c80af3a0 con 0x7fc5c80ad6c0 2024-03-29T10:42:41.024 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:41.036+0000 7fc5c559a640 10 monclient(hunting): _negotiate 2024-03-29T10:42:41.024 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:41.036+0000 7fc5c559a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:42:41.024 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:41.036+0000 7fc5c559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:42:41.024 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:41.036+0000 7fc5c559a640 10 monclient(hunting): my global_id is 4291 2024-03-29T10:42:41.024 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:41.036+0000 7fc5c559a640 1 -- 172.21.15.88:0/4168542110 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc5c80afb60 con 0x7fc5c80ad6c0 2024-03-29T10:42:41.024 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:41.037+0000 7fc5c559a640 1 -- 172.21.15.88:0/4168542110 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 175304155 0 0) 0x7fc5b0003520 con 0x7fc5c814fdc0 2024-03-29T10:42:41.024 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:41.037+0000 7fc5c559a640 1 -- 172.21.15.88:0/4168542110 >> v1:172.21.15.196:6789/0 conn(0x7fc5c80ad6c0 legacy=0x7fc5c814f6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:41.024 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:41.037+0000 7fc5c559a640 1 -- 172.21.15.88:0/4168542110 >> v1:172.21.15.88:6789/0 conn(0x7fc5c80af6c0 legacy=0x7fc5c8152980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:42:41.025 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:41.037+0000 7fc5c559a640 10 monclient: _finish_hunting 0 2024-03-29T10:42:41.025 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:41.037+0000 7fc5c559a640 1 monclient: found mon.noname-b 2024-03-29T10:42:41.025 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:41.037+0000 7fc5c559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:42:41.025 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:41.037+0000 7fc5c559a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.88:6790/0 2024-03-29T10:42:41.025 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:41.037+0000 7fc5c559a640 1 -- 172.21.15.88:0/4168542110 --> v1:172.21.15.88:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5c80aab80 con 0x7fc5c814fdc0 2024-03-29T10:42:41.025 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:41.037+0000 7fc5c559a640 10 monclient: _finish_auth 0 2024-03-29T10:42:41.025 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:41.037+0000 7fc5c559a640 10 monclient: _check_auth_tickets 2024-03-29T10:42:41.025 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:41.037+0000 7fc5c559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:42:41.025 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:41.037+0000 7fc5c559a640 1 -- 172.21.15.88:0/4168542110 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7fc5b8001690 con 0x7fc5c814fdc0 2024-03-29T10:42:41.025 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:41.037+0000 7fc5c559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:41.025 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:41.037+0000 7fc5c559a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:42:41.025 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:41.037+0000 7fc5c559a640 10 monclient: dump: 2024-03-29T10:42:41.025 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:42:41.025 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:42:41.025 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:42:41.026 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:42:41.026 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:41.026 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:42:41.026 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:42:41.026 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:42:41.026 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:42:41.026 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:42:41.026 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:41.038+0000 7fc5c559a640 1 -- 172.21.15.88:0/4168542110 <== mon.2 v1:172.21.15.88:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc5b80017f0 con 0x7fc5c814fdc0 2024-03-29T10:42:41.026 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:41.038+0000 7fc5c559a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:42:41.026 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:41.038+0000 7fc5c559a640 1 -- 172.21.15.88:0/4168542110 <== mon.2 v1:172.21.15.88:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7fc5b8001c00 con 0x7fc5c814fdc0 2024-03-29T10:42:41.026 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:41.038+0000 7fc5c559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:41.026 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:41.038+0000 7fc5c559a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:42:41.026 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:41.038+0000 7fc5c559a640 10 monclient: dump: 2024-03-29T10:42:41.026 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:42:41.027 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:42:41.027 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:42:41.027 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:42:41.027 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:41.027 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:42:41.027 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:42:41.027 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:42:41.027 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:42:41.027 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:42:41.027 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:41.038+0000 7fc5cce06640 5 monclient: authenticate success, global_id 4307 2024-03-29T10:42:41.027 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:41.038+0000 7fc5cce06640 10 monclient: get_monmap_and_config success 2024-03-29T10:42:41.027 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:41.038+0000 7fc5cce06640 4 set_mon_vals no callback set 2024-03-29T10:42:41.027 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:41.039+0000 7fc5cce06640 10 monclient: shutdown 2024-03-29T10:42:41.028 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:41.039+0000 7fc5cce06640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:42:41.028 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:41.039+0000 7fc5cce06640 1 -- 172.21.15.88:0/4168542110 >> v1:172.21.15.88:6790/0 conn(0x7fc5c814fdc0 legacy=0x7fc5c815a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:41.028 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:41.039+0000 7fc5cce06640 1 -- 172.21.15.88:0/4168542110 shutdown_connections 2024-03-29T10:42:41.028 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:41.039+0000 7fc5cce06640 1 -- 172.21.15.88:0/4168542110 >> 172.21.15.88:0/4168542110 conn(0x7fc5c80b35e0 msgr2=0x7fc5c80b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:42:41.028 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:41.039+0000 7fc5cce06640 1 -- 172.21.15.88:0/4168542110 shutdown_connections 2024-03-29T10:42:41.028 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:41.040+0000 7fc5cce06640 1 -- 172.21.15.88:0/4168542110 wait complete. 2024-03-29T10:42:41.028 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:41.041+0000 7fc5cce06640 10 monclient: build_initial_monmap 2024-03-29T10:42:41.028 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:41.041+0000 7fc5cce06640 10 monclient: monmap: 2024-03-29T10:42:41.029 INFO:teuthology.misc.health.smithi088.stderr:epoch 0 2024-03-29T10:42:41.029 INFO:teuthology.misc.health.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:42:41.029 INFO:teuthology.misc.health.smithi088.stderr:last_changed 0.000000 2024-03-29T10:42:41.029 INFO:teuthology.misc.health.smithi088.stderr:created 0.000000 2024-03-29T10:42:41.029 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 0 (unknown) 2024-03-29T10:42:41.029 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:42:41.029 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-03-29T10:42:41.029 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-29T10:42:41.029 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-c 2024-03-29T10:42:41.029 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:42:41.029 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:41.041+0000 7fc5cce06640 1 librados: starting msgr at 2024-03-29T10:42:41.029 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:41.041+0000 7fc5cce06640 1 librados: starting objecter 2024-03-29T10:42:41.029 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:41.041+0000 7fc5cce06640 1 Processor -- start 2024-03-29T10:42:41.029 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:41.041+0000 7fc5cce06640 1 -- start start 2024-03-29T10:42:41.030 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:41.041+0000 7fc5cce06640 1 librados: setting wanted keys 2024-03-29T10:42:41.030 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:41.041+0000 7fc5cce06640 1 librados: calling monclient init 2024-03-29T10:42:41.030 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:41.041+0000 7fc5cce06640 10 monclient: init 2024-03-29T10:42:41.030 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:41.042+0000 7fc5cce06640 10 monclient: _reopen_session rank -1 2024-03-29T10:42:41.030 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:41.042+0000 7fc5cce06640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:42:41.030 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:41.042+0000 7fc5cce06640 10 monclient(hunting): picked mon.noname-b con 0x7fc5c80ad6c0 addr v1:172.21.15.196:6789/0 2024-03-29T10:42:41.030 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:41.042+0000 7fc5cce06640 10 monclient(hunting): picked mon.noname-c con 0x7fc5c80af6c0 addr v1:172.21.15.88:6790/0 2024-03-29T10:42:41.030 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:41.042+0000 7fc5cce06640 10 monclient(hunting): picked mon.noname-a con 0x7fc5c814fdc0 addr v1:172.21.15.88:6789/0 2024-03-29T10:42:41.030 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:41.042+0000 7fc5cce06640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5c81e4560 con 0x7fc5c814fdc0 2024-03-29T10:42:41.030 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:41.042+0000 7fc5cce06640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5c806eeb0 con 0x7fc5c80ad6c0 2024-03-29T10:42:41.030 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:41.042+0000 7fc5cce06640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5c81e4a80 con 0x7fc5c80af6c0 2024-03-29T10:42:41.030 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:41.042+0000 7fc5cce06640 10 monclient(hunting): _renew_subs 2024-03-29T10:42:41.030 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:41.042+0000 7fc5cce06640 10 monclient(hunting): authenticate will time out at 883.278658s 2024-03-29T10:42:41.031 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:41.047+0000 7fc5c6d9d640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fc5c814fdc0 0x7fc5c81ef6d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:46365/0 (socket says 172.21.15.88:46365) 2024-03-29T10:42:41.031 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:41.047+0000 7fc5c6d9d640 1 -- 172.21.15.88:0/1714669277 learned_addr learned my addr 172.21.15.88:0/1714669277 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-29T10:42:41.031 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:41.048+0000 7fc5b6ffd640 10 client.?.objecter ms_handle_connect 0x7fc5c814fdc0 2024-03-29T10:42:41.031 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:41.048+0000 7fc5b6ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:42:41.031 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:41.048+0000 7fc5b6ffd640 1 -- 172.21.15.88:0/1714669277 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1510394030 0 0) 0x7fc5c81e4560 con 0x7fc5c814fdc0 2024-03-29T10:42:41.031 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:41.048+0000 7fc5b6ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:42:41.031 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:41.048+0000 7fc5b6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:42:41.032 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:41.048+0000 7fc5b6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:42:41.032 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:41.048+0000 7fc5b6ffd640 10 monclient(hunting): my global_id is 4302 2024-03-29T10:42:41.032 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:41.048+0000 7fc5b6ffd640 1 -- 172.21.15.88:0/1714669277 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc5a8004eb0 con 0x7fc5c814fdc0 2024-03-29T10:42:41.032 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:41.049+0000 7fc5b6ffd640 1 -- 172.21.15.88:0/1714669277 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1378567375 0 0) 0x7fc5a8004eb0 con 0x7fc5c814fdc0 2024-03-29T10:42:41.032 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:41.049+0000 7fc5b6ffd640 1 -- 172.21.15.88:0/1714669277 --> v1:172.21.15.88:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc5c81e4560 con 0x7fc5c814fdc0 2024-03-29T10:42:41.032 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:41.049+0000 7fc5b6ffd640 1 -- 172.21.15.88:0/1714669277 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7fc5b8001c00 con 0x7fc5c814fdc0 2024-03-29T10:42:41.033 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:41.049+0000 7fc5b6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:41.033 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:41.049+0000 7fc5b6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:42:41.033 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:41.049+0000 7fc5b6ffd640 10 monclient(hunting): dump: 2024-03-29T10:42:41.033 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:42:41.033 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:42:41.033 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:42:41.033 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:42:41.033 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:41.033 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:42:41.033 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:42:41.033 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:42:41.033 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:42:41.033 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:42:41.033 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:41.049+0000 7fc5b6ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:42:41.034 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:41.049+0000 7fc5b6ffd640 10 client.?.objecter ms_handle_connect 0x7fc5c80ad6c0 2024-03-29T10:42:41.034 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:41.049+0000 7fc5b6ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:42:41.034 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:41.051+0000 7fc5b6ffd640 10 client.?.objecter ms_handle_connect 0x7fc5c80af6c0 2024-03-29T10:42:41.034 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:41.051+0000 7fc5b6ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:42:41.034 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:41.051+0000 7fc5b6ffd640 1 -- 172.21.15.88:0/1714669277 <== mon.0 v1:172.21.15.88:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3925601204 0 0) 0x7fc5c81e4560 con 0x7fc5c814fdc0 2024-03-29T10:42:41.035 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:41.051+0000 7fc5b6ffd640 1 -- 172.21.15.88:0/1714669277 >> v1:172.21.15.88:6790/0 conn(0x7fc5c80af6c0 legacy=0x7fc5c814d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:41.035 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:41.051+0000 7fc5b6ffd640 1 -- 172.21.15.88:0/1714669277 >> v1:172.21.15.196:6789/0 conn(0x7fc5c80ad6c0 legacy=0x7fc5c806e7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:41.035 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:41.051+0000 7fc5b6ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:42:41.035 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:41.051+0000 7fc5b6ffd640 1 monclient: found mon.a 2024-03-29T10:42:41.035 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:41.051+0000 7fc5b6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:42:41.035 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:41.051+0000 7fc5b6ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-03-29T10:42:41.035 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:41.051+0000 7fc5b6ffd640 1 -- 172.21.15.88:0/1714669277 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5c81e4c10 con 0x7fc5c814fdc0 2024-03-29T10:42:41.035 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:41.051+0000 7fc5b6ffd640 10 monclient: _finish_auth 0 2024-03-29T10:42:41.035 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:41.051+0000 7fc5b6ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:42:41.036 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:41.051+0000 7fc5b6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:42:41.036 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:41.051+0000 7fc5cce06640 5 monclient: authenticate success, global_id 4302 2024-03-29T10:42:41.036 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:41.051+0000 7fc5cce06640 10 monclient: _renew_subs 2024-03-29T10:42:41.036 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:41.051+0000 7fc5cce06640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-03-29T10:42:41.036 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:41.051+0000 7fc5cce06640 1 -- 172.21.15.88:0/1714669277 --> v1:172.21.15.88:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5c814dda0 con 0x7fc5c814fdc0 2024-03-29T10:42:41.036 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:41.051+0000 7fc5cce06640 10 client.4302.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:42:41.036 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:41.051+0000 7fc5cce06640 10 monclient: _renew_subs 2024-03-29T10:42:41.036 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:41.051+0000 7fc5cce06640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-03-29T10:42:41.036 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:41.051+0000 7fc5cce06640 1 -- 172.21.15.88:0/1714669277 --> v1:172.21.15.88:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5c808a760 con 0x7fc5c814fdc0 2024-03-29T10:42:41.036 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:41.051+0000 7fc5cce06640 1 librados: init done 2024-03-29T10:42:41.037 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:41.052+0000 7fc5b6ffd640 1 -- 172.21.15.88:0/1714669277 <== mon.0 v1:172.21.15.88:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc5b80038a0 con 0x7fc5c814fdc0 2024-03-29T10:42:41.037 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:41.052+0000 7fc5b6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:42:41.037 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:41.052+0000 7fc5b7fff640 4 set_mon_vals no callback set 2024-03-29T10:42:41.037 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:41.053+0000 7fc5cce06640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:42:41.038 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:41.053+0000 7fc5b6ffd640 1 -- 172.21.15.88:0/1714669277 <== mon.0 v1:172.21.15.88:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7fc5b8003cf0 con 0x7fc5c814fdc0 2024-03-29T10:42:41.038 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:41.053+0000 7fc5b6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:41.038 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:41.053+0000 7fc5b6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:42:41.038 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:41.053+0000 7fc5b6ffd640 10 monclient: dump: 2024-03-29T10:42:41.038 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:42:41.038 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:42:41.038 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:42:41.038 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:42:41.038 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:41.038 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:42:41.038 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:42:41.038 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:42:41.038 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:42:41.038 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:42:41.039 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:41.054+0000 7fc5cce06640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:42:41.040 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:41.054+0000 7fc5cce06640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-03-29T10:42:41.040 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:41.054+0000 7fc5cce06640 1 -- 172.21.15.88:0/1714669277 --> v1:172.21.15.88:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5c81e46f0 con 0x7fc5c814fdc0 2024-03-29T10:42:41.040 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:41.055+0000 7fc5b6ffd640 1 -- 172.21.15.88:0/1714669277 <== mon.0 v1:172.21.15.88:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4235629220 0 0) 0x7fc5b8003e50 con 0x7fc5c814fdc0 2024-03-29T10:42:41.040 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:41.056+0000 7fc5b6ffd640 1 -- 172.21.15.88:0/1714669277 <== mon.0 v1:172.21.15.88:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4257673582 0 0) 0x7fc5b8084b30 con 0x7fc5c814fdc0 2024-03-29T10:42:41.040 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:41.056+0000 7fc5b6ffd640 10 client.4302.objecter ms_dispatch 0x7fc5c8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:42:41.040 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:41.056+0000 7fc5b6ffd640 3 client.4302.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:42:41.040 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:41.056+0000 7fc5b6ffd640 3 client.4302.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:42:41.040 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:41.056+0000 7fc5b6ffd640 20 client.4302.objecter dump_active .. 0 homeless 2024-03-29T10:42:41.041 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:41.058+0000 7fc5c659c640 10 monclient: get_auth_request con 0x7fc5a806dbd0 auth_method 0 2024-03-29T10:42:41.042 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:41.059+0000 7fc5b6ffd640 10 client.4302.objecter ms_handle_connect 0x7fc5a806dbd0 2024-03-29T10:42:41.050 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:41.067+0000 7fc5b6ffd640 1 -- 172.21.15.88:0/1714669277 <== mon.0 v1:172.21.15.88:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc5b8050410 con 0x7fc5c814fdc0 2024-03-29T10:42:41.050 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:41.067+0000 7fc5b6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:42:41.050 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:41.067+0000 7fc5b6ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:42:41.379 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:41.393+0000 7fc5cce06640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:42:41.379 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:41.393+0000 7fc5cce06640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:42:41.379 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:41.393+0000 7fc5cce06640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-03-29T10:42:41.379 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:41.393+0000 7fc5cce06640 1 -- 172.21.15.88:0/1714669277 --> v1:172.21.15.88:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc5c8075f10 con 0x7fc5c814fdc0 2024-03-29T10:42:41.379 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:41.395+0000 7fc5b6ffd640 1 -- 172.21.15.88:0/1714669277 <== mon.0 v1:172.21.15.88:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 344506374) 0x7fc5b8052340 con 0x7fc5c814fdc0 2024-03-29T10:42:41.379 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:41.395+0000 7fc5b6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:42:41.379 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:41.395+0000 7fc5b6ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:42:41.380 INFO:teuthology.misc.health.smithi088.stdout: 2024-03-29T10:42:41.380 INFO:teuthology.misc.health.smithi088.stdout:{"epoch":10,"fsid":"869f5188-f92b-42dd-a46d-8c646e0388a9","created":"2024-03-29T10:40:40.285087+0000","modified":"2024-03-29T10:40:42.536230+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:40:42.536230+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a26c0c1c-8de0-438c-a3f6-3df10ea61574","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5de24915-8dec-49b6-8b16-f3170197c591","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ac054af3-d589-4565-9644-e6af75975c53","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3ec4f045-c714-4a22-bd1a-e07330b45ab3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"cddb9867-8e30-4951-9b13-3d1e78b9c45a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dc6d9911-785d-432f-8b53-56a1ccab302f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"77c88437-b587-496e-8b4d-1824ea8b5305","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ce963d96-5a8e-44c3-99de-c0dbebe234ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:42:41.383 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:41.400+0000 7fc5cce06640 10 librados: watch_flush enter 2024-03-29T10:42:41.384 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:41.400+0000 7fc5cce06640 10 librados: watch_flush exit 2024-03-29T10:42:41.384 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:41.400+0000 7fc5cce06640 20 client.4302.objecter shutdown clearing up homeless session... 2024-03-29T10:42:41.384 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:41.400+0000 7fc5cce06640 10 client.4302.objecter successfully canceled tick 2024-03-29T10:42:41.384 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:41.401+0000 7fc5cce06640 1 -- 172.21.15.88:0/1714669277 >> v1:172.21.15.88:6816/1752861349 conn(0x7fc5a806dbd0 legacy=0x7fc5a8070090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:41.384 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:41.401+0000 7fc5cce06640 10 monclient: shutdown 2024-03-29T10:42:41.384 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:41.401+0000 7fc5cce06640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:42:41.384 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:41.401+0000 7fc5cce06640 1 -- 172.21.15.88:0/1714669277 >> v1:172.21.15.88:6789/0 conn(0x7fc5c814fdc0 legacy=0x7fc5c81ef6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:41.385 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:41.402+0000 7fc5cce06640 1 -- 172.21.15.88:0/1714669277 shutdown_connections 2024-03-29T10:42:41.385 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:41.402+0000 7fc5cce06640 1 -- 172.21.15.88:0/1714669277 >> 172.21.15.88:0/1714669277 conn(0x7fc5c80b35e0 msgr2=0x7fc5c80ac140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:42:41.385 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:41.402+0000 7fc5cce06640 1 -- 172.21.15.88:0/1714669277 shutdown_connections 2024-03-29T10:42:41.385 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:41.402+0000 7fc5cce06640 1 -- 172.21.15.88:0/1714669277 wait complete. 2024-03-29T10:42:41.386 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:41.402+0000 7fc5cce06640 1 librados: shutdown 2024-03-29T10:42:41.402 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:42:48.209 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:42:48.364 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:48.380+0000 7ffbb86f4640 10 monclient: get_monmap_and_config 2024-03-29T10:42:48.365 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:48.380+0000 7ffbb86f4640 10 monclient: build_initial_monmap 2024-03-29T10:42:48.365 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:48.380+0000 7ffbb86f4640 10 monclient: monmap: 2024-03-29T10:42:48.365 INFO:teuthology.misc.health.smithi088.stderr:epoch 0 2024-03-29T10:42:48.365 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:42:48.365 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:42:48.381829+0000 2024-03-29T10:42:48.365 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:42:48.381829+0000 2024-03-29T10:42:48.365 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 0 (unknown) 2024-03-29T10:42:48.365 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:42:48.365 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-03-29T10:42:48.365 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-29T10:42:48.365 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-b 2024-03-29T10:42:48.365 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:42:48.366 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:48.382+0000 7ffbb86f4640 1 Processor -- start 2024-03-29T10:42:48.366 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:48.382+0000 7ffbb86f4640 1 -- start start 2024-03-29T10:42:48.366 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:48.382+0000 7ffbb86f4640 10 monclient: init 2024-03-29T10:42:48.371 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:48.384+0000 7ffbb86f4640 10 monclient: _reopen_session rank -1 2024-03-29T10:42:48.371 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:48.384+0000 7ffbb86f4640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:42:48.371 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:48.384+0000 7ffbb86f4640 10 monclient(hunting): picked mon.noname-a con 0x7ffbb00a9750 addr v1:172.21.15.88:6789/0 2024-03-29T10:42:48.371 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:48.384+0000 7ffbb86f4640 10 monclient(hunting): picked mon.noname-b con 0x7ffbb0149890 addr v1:172.21.15.88:6790/0 2024-03-29T10:42:48.371 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:48.384+0000 7ffbb86f4640 10 monclient(hunting): picked mon.noname-c con 0x7ffbb00afa30 addr v1:172.21.15.196:6789/0 2024-03-29T10:42:48.371 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:48.384+0000 7ffbb86f4640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffbb00b00c0 con 0x7ffbb00a9750 2024-03-29T10:42:48.371 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:48.384+0000 7ffbb86f4640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffbb01464a0 con 0x7ffbb00afa30 2024-03-29T10:42:48.371 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:48.384+0000 7ffbb86f4640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffbb00ab280 con 0x7ffbb0149890 2024-03-29T10:42:48.371 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:48.384+0000 7ffbb86f4640 10 monclient(hunting): _renew_subs 2024-03-29T10:42:48.371 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:48.384+0000 7ffbb86f4640 10 monclient(hunting): authenticate will time out at 890.620155s 2024-03-29T10:42:48.371 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:48.387+0000 7ffbb6490640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7ffbb00a9750 0x7ffbb01469a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:54629/0 (socket says 172.21.15.88:54629) 2024-03-29T10:42:48.372 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:48.387+0000 7ffbb6490640 1 -- 172.21.15.88:0/1829611005 learned_addr learned my addr 172.21.15.88:0/1829611005 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-29T10:42:48.372 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:48.388+0000 7ffbb548e640 1 -- 172.21.15.88:0/1829611005 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 747405754 0 0) 0x7ffbb00b00c0 con 0x7ffbb00a9750 2024-03-29T10:42:48.372 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:48.388+0000 7ffbb548e640 10 monclient(hunting): _negotiate 2024-03-29T10:42:48.372 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:48.388+0000 7ffbb548e640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:42:48.372 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:48.388+0000 7ffbb548e640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:42:48.372 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:48.388+0000 7ffbb548e640 10 monclient(hunting): my global_id is 4305 2024-03-29T10:42:48.372 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:48.388+0000 7ffbb548e640 1 -- 172.21.15.88:0/1829611005 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb9c003520 con 0x7ffbb00a9750 2024-03-29T10:42:48.372 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:48.388+0000 7ffbb548e640 1 -- 172.21.15.88:0/1829611005 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2241811122 0 0) 0x7ffb9c003520 con 0x7ffbb00a9750 2024-03-29T10:42:48.372 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:48.388+0000 7ffbb548e640 1 -- 172.21.15.88:0/1829611005 >> v1:172.21.15.88:6790/0 conn(0x7ffbb0149890 legacy=0x7ffbb0149da0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:42:48.373 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:48.390+0000 7ffbb548e640 1 -- 172.21.15.88:0/1829611005 >> v1:172.21.15.196:6789/0 conn(0x7ffbb00afa30 legacy=0x7ffbb0151c30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:42:48.375 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:48.391+0000 7ffbb548e640 10 monclient: _finish_hunting 0 2024-03-29T10:42:48.375 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:48.391+0000 7ffbb548e640 1 monclient: found mon.noname-a 2024-03-29T10:42:48.376 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:48.391+0000 7ffbb548e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:42:48.376 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:48.391+0000 7ffbb548e640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.88:6789/0 2024-03-29T10:42:48.376 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:48.391+0000 7ffbb548e640 1 -- 172.21.15.88:0/1829611005 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbb01470b0 con 0x7ffbb00a9750 2024-03-29T10:42:48.376 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:48.391+0000 7ffbb548e640 10 monclient: _finish_auth 0 2024-03-29T10:42:48.376 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:48.391+0000 7ffbb548e640 10 monclient: _check_auth_tickets 2024-03-29T10:42:48.376 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:48.391+0000 7ffbb548e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:42:48.376 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:48.391+0000 7ffbb86f4640 5 monclient: authenticate success, global_id 4305 2024-03-29T10:42:48.376 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:48.391+0000 7ffbb86f4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:42:48.376 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:48.391+0000 7ffbb548e640 1 -- 172.21.15.88:0/1829611005 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7ffba8001a10 con 0x7ffbb00a9750 2024-03-29T10:42:48.376 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:48.391+0000 7ffbb548e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:48.376 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:48.391+0000 7ffbb548e640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:42:48.377 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:48.391+0000 7ffbb548e640 10 monclient: dump: 2024-03-29T10:42:48.378 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:42:48.378 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:42:48.378 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:42:48.378 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:42:48.378 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:48.378 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:42:48.378 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:42:48.378 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:42:48.378 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:42:48.378 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:42:48.378 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:48.392+0000 7ffbb86f4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:42:48.378 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:48.392+0000 7ffbb548e640 1 -- 172.21.15.88:0/1829611005 <== mon.0 v1:172.21.15.88:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ffba8001ca0 con 0x7ffbb00a9750 2024-03-29T10:42:48.379 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:48.392+0000 7ffbb548e640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:42:48.379 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:48.392+0000 7ffbb86f4640 10 monclient: get_monmap_and_config success 2024-03-29T10:42:48.379 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:48.392+0000 7ffbb86f4640 4 set_mon_vals no callback set 2024-03-29T10:42:48.379 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:48.393+0000 7ffbb86f4640 10 monclient: shutdown 2024-03-29T10:42:48.379 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:48.393+0000 7ffbb86f4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:42:48.379 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:48.393+0000 7ffbb86f4640 1 -- 172.21.15.88:0/1829611005 >> v1:172.21.15.88:6789/0 conn(0x7ffbb00a9750 legacy=0x7ffbb01469a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:48.381 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:48.395+0000 7ffbb86f4640 1 -- 172.21.15.88:0/1829611005 shutdown_connections 2024-03-29T10:42:48.381 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:48.395+0000 7ffbb86f4640 1 -- 172.21.15.88:0/1829611005 >> 172.21.15.88:0/1829611005 conn(0x7ffbb00b3a00 msgr2=0x7ffbb00b3de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:42:48.381 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:48.395+0000 7ffbb86f4640 1 -- 172.21.15.88:0/1829611005 shutdown_connections 2024-03-29T10:42:48.381 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:48.395+0000 7ffbb86f4640 1 -- 172.21.15.88:0/1829611005 wait complete. 2024-03-29T10:42:48.381 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:48.396+0000 7ffbb86f4640 10 monclient: build_initial_monmap 2024-03-29T10:42:48.381 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:48.396+0000 7ffbb86f4640 10 monclient: monmap: 2024-03-29T10:42:48.381 INFO:teuthology.misc.health.smithi088.stderr:epoch 0 2024-03-29T10:42:48.381 INFO:teuthology.misc.health.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:42:48.381 INFO:teuthology.misc.health.smithi088.stderr:last_changed 0.000000 2024-03-29T10:42:48.381 INFO:teuthology.misc.health.smithi088.stderr:created 0.000000 2024-03-29T10:42:48.381 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 0 (unknown) 2024-03-29T10:42:48.381 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:42:48.382 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-03-29T10:42:48.382 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-29T10:42:48.382 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-c 2024-03-29T10:42:48.382 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:42:48.382 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:48.397+0000 7ffbb86f4640 1 librados: starting msgr at 2024-03-29T10:42:48.382 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:48.397+0000 7ffbb86f4640 1 librados: starting objecter 2024-03-29T10:42:48.382 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:48.397+0000 7ffbb86f4640 1 Processor -- start 2024-03-29T10:42:48.384 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:48.397+0000 7ffbb86f4640 1 -- start start 2024-03-29T10:42:48.384 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:48.397+0000 7ffbb86f4640 1 librados: setting wanted keys 2024-03-29T10:42:48.384 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:48.397+0000 7ffbb86f4640 1 librados: calling monclient init 2024-03-29T10:42:48.384 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:48.397+0000 7ffbb86f4640 10 monclient: init 2024-03-29T10:42:48.384 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:48.398+0000 7ffbb86f4640 10 monclient: _reopen_session rank -1 2024-03-29T10:42:48.384 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:48.398+0000 7ffbb86f4640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:42:48.384 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:48.398+0000 7ffbb86f4640 10 monclient(hunting): picked mon.noname-c con 0x7ffbb00a9750 addr v1:172.21.15.88:6790/0 2024-03-29T10:42:48.384 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:48.398+0000 7ffbb86f4640 10 monclient(hunting): picked mon.noname-b con 0x7ffbb00afa30 addr v1:172.21.15.196:6789/0 2024-03-29T10:42:48.384 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:48.398+0000 7ffbb86f4640 10 monclient(hunting): picked mon.noname-a con 0x7ffbb0149890 addr v1:172.21.15.88:6789/0 2024-03-29T10:42:48.384 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:48.398+0000 7ffbb86f4640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffbb01db350 con 0x7ffbb0149890 2024-03-29T10:42:48.384 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:48.398+0000 7ffbb86f4640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffbb0140c60 con 0x7ffbb00afa30 2024-03-29T10:42:48.384 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:48.398+0000 7ffbb86f4640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffbb006d8c0 con 0x7ffbb00a9750 2024-03-29T10:42:48.384 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:48.398+0000 7ffbb86f4640 10 monclient(hunting): _renew_subs 2024-03-29T10:42:48.385 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:48.398+0000 7ffbb86f4640 10 monclient(hunting): authenticate will time out at 890.634569s 2024-03-29T10:42:48.388 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:48.401+0000 7ffbb6490640 1 --1- >> v1:172.21.15.88:6790/0 conn(0x7ffbb00a9750 0x7ffbb01db7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6790/0 says I am v1:172.21.15.88:59277/0 (socket says 172.21.15.88:59277) 2024-03-29T10:42:48.388 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:48.401+0000 7ffbb6490640 1 -- 172.21.15.88:0/1022522417 learned_addr learned my addr 172.21.15.88:0/1022522417 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-29T10:42:48.388 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:48.401+0000 7ffba6ffd640 10 client.?.objecter ms_handle_connect 0x7ffbb00a9750 2024-03-29T10:42:48.388 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:48.401+0000 7ffba6ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:42:48.388 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:48.401+0000 7ffba6ffd640 1 -- 172.21.15.88:0/1022522417 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1734228999 0 0) 0x7ffbb006d8c0 con 0x7ffbb00a9750 2024-03-29T10:42:48.388 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:48.401+0000 7ffba6ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:42:48.388 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:48.401+0000 7ffba6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:42:48.388 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:48.401+0000 7ffba6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:42:48.388 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:48.401+0000 7ffba6ffd640 10 monclient(hunting): my global_id is 4313 2024-03-29T10:42:48.389 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:48.402+0000 7ffba6ffd640 1 -- 172.21.15.88:0/1022522417 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb94004eb0 con 0x7ffbb00a9750 2024-03-29T10:42:48.389 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:48.402+0000 7ffba6ffd640 1 -- 172.21.15.88:0/1022522417 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1341096314 0 0) 0x7ffb94004eb0 con 0x7ffbb00a9750 2024-03-29T10:42:48.389 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:48.402+0000 7ffba6ffd640 1 -- 172.21.15.88:0/1022522417 --> v1:172.21.15.88:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ffbb006d8c0 con 0x7ffbb00a9750 2024-03-29T10:42:48.389 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:48.402+0000 7ffba6ffd640 1 -- 172.21.15.88:0/1022522417 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7ffba8004e60 con 0x7ffbb00a9750 2024-03-29T10:42:48.389 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:48.402+0000 7ffba6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:48.389 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:48.402+0000 7ffba6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:42:48.389 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:48.402+0000 7ffba6ffd640 10 monclient(hunting): dump: 2024-03-29T10:42:48.389 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:42:48.389 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:42:48.389 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:42:48.389 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:42:48.389 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:48.389 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:42:48.389 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:42:48.390 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:42:48.390 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:42:48.390 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:42:48.390 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:48.402+0000 7ffba6ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:42:48.390 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:48.403+0000 7ffba6ffd640 1 -- 172.21.15.88:0/1022522417 <== mon.2 v1:172.21.15.88:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 321212208 0 0) 0x7ffbb006d8c0 con 0x7ffbb00a9750 2024-03-29T10:42:48.390 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:48.403+0000 7ffba6ffd640 1 -- 172.21.15.88:0/1022522417 >> v1:172.21.15.196:6789/0 conn(0x7ffbb00afa30 legacy=0x7ffbb013fe40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:42:48.390 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:48.403+0000 7ffba6ffd640 1 -- 172.21.15.88:0/1022522417 >> v1:172.21.15.88:6789/0 conn(0x7ffbb0149890 legacy=0x7ffbb0140550 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:42:48.390 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:48.405+0000 7ffba6ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:42:48.390 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:48.405+0000 7ffba6ffd640 1 monclient: found mon.c 2024-03-29T10:42:48.391 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:48.405+0000 7ffba6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:42:48.391 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:48.405+0000 7ffba6ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-03-29T10:42:48.391 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:48.405+0000 7ffba6ffd640 1 -- 172.21.15.88:0/1022522417 --> v1:172.21.15.88:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbb00723f0 con 0x7ffbb00a9750 2024-03-29T10:42:48.391 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:48.405+0000 7ffba6ffd640 10 monclient: _finish_auth 0 2024-03-29T10:42:48.391 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:48.405+0000 7ffba6ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:42:48.391 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:48.405+0000 7ffba6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:42:48.391 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:48.405+0000 7ffbb86f4640 5 monclient: authenticate success, global_id 4313 2024-03-29T10:42:48.391 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:48.405+0000 7ffbb86f4640 10 monclient: _renew_subs 2024-03-29T10:42:48.391 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:48.405+0000 7ffbb86f4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-03-29T10:42:48.391 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:48.405+0000 7ffbb86f4640 1 -- 172.21.15.88:0/1022522417 --> v1:172.21.15.88:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffbb0070870 con 0x7ffbb00a9750 2024-03-29T10:42:48.391 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:48.405+0000 7ffbb86f4640 10 client.4313.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:42:48.391 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:48.405+0000 7ffbb86f4640 10 monclient: _renew_subs 2024-03-29T10:42:48.391 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:48.405+0000 7ffbb86f4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-03-29T10:42:48.391 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:48.405+0000 7ffbb86f4640 1 -- 172.21.15.88:0/1022522417 --> v1:172.21.15.88:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ffbb00713a0 con 0x7ffbb00a9750 2024-03-29T10:42:48.392 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:48.405+0000 7ffbb86f4640 1 librados: init done 2024-03-29T10:42:48.394 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:48.409+0000 7ffbb86f4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:42:48.394 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:48.409+0000 7ffbb86f4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:42:48.394 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:48.409+0000 7ffbb86f4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-03-29T10:42:48.394 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:48.409+0000 7ffba6ffd640 1 -- 172.21.15.88:0/1022522417 <== mon.2 v1:172.21.15.88:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ffba8001e90 con 0x7ffbb00a9750 2024-03-29T10:42:48.394 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:48.409+0000 7ffbb86f4640 1 -- 172.21.15.88:0/1022522417 --> v1:172.21.15.88:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffbb00adf50 con 0x7ffbb00a9750 2024-03-29T10:42:48.394 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:48.409+0000 7ffba6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:42:48.394 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:48.409+0000 7ffba6ffd640 1 -- 172.21.15.88:0/1022522417 <== mon.2 v1:172.21.15.88:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7ffba8005410 con 0x7ffbb00a9750 2024-03-29T10:42:48.394 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:48.409+0000 7ffba6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:48.394 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:48.409+0000 7ffba6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:42:48.394 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:48.409+0000 7ffba6ffd640 10 monclient: dump: 2024-03-29T10:42:48.395 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:42:48.395 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:42:48.395 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:42:48.395 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:42:48.395 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:48.395 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:42:48.395 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:42:48.395 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:42:48.396 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:42:48.396 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:42:48.396 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:48.409+0000 7ffbb4c8d640 4 set_mon_vals no callback set 2024-03-29T10:42:48.396 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:48.410+0000 7ffba6ffd640 1 -- 172.21.15.88:0/1022522417 <== mon.2 v1:172.21.15.88:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4235629220 0 0) 0x7ffba8005630 con 0x7ffbb00a9750 2024-03-29T10:42:48.396 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:48.411+0000 7ffba6ffd640 1 -- 172.21.15.88:0/1022522417 <== mon.2 v1:172.21.15.88:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4257673582 0 0) 0x7ffba8085250 con 0x7ffbb00a9750 2024-03-29T10:42:48.396 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:48.411+0000 7ffba6ffd640 10 client.4313.objecter ms_dispatch 0x7ffbb0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:42:48.396 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:48.411+0000 7ffba6ffd640 3 client.4313.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:42:48.397 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:48.411+0000 7ffba6ffd640 3 client.4313.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:42:48.397 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:48.411+0000 7ffba6ffd640 20 client.4313.objecter dump_active .. 0 homeless 2024-03-29T10:42:48.397 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:48.411+0000 7ffbb5c8f640 10 monclient: get_auth_request con 0x7ffb9406dbb0 auth_method 0 2024-03-29T10:42:48.397 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:48.412+0000 7ffba6ffd640 10 client.4313.objecter ms_handle_connect 0x7ffb9406dbb0 2024-03-29T10:42:48.406 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:48.422+0000 7ffba6ffd640 1 -- 172.21.15.88:0/1022522417 <== mon.2 v1:172.21.15.88:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ffba8050b30 con 0x7ffbb00a9750 2024-03-29T10:42:48.406 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:48.422+0000 7ffba6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:42:48.406 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:48.422+0000 7ffba6ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:42:48.745 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:48.762+0000 7ffbb86f4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:42:48.745 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:48.762+0000 7ffbb86f4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:42:48.745 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:48.762+0000 7ffbb86f4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-03-29T10:42:48.745 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:48.762+0000 7ffbb86f4640 1 -- 172.21.15.88:0/1022522417 --> v1:172.21.15.88:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ffbb004f730 con 0x7ffbb00a9750 2024-03-29T10:42:48.746 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:48.763+0000 7ffba6ffd640 1 -- 172.21.15.88:0/1022522417 <== mon.2 v1:172.21.15.88:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 344506374) 0x7ffba8052a60 con 0x7ffbb00a9750 2024-03-29T10:42:48.746 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:48.763+0000 7ffba6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:42:48.746 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:48.763+0000 7ffba6ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:42:48.746 INFO:teuthology.misc.health.smithi088.stdout: 2024-03-29T10:42:48.746 INFO:teuthology.misc.health.smithi088.stdout:{"epoch":10,"fsid":"869f5188-f92b-42dd-a46d-8c646e0388a9","created":"2024-03-29T10:40:40.285087+0000","modified":"2024-03-29T10:40:42.536230+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:40:42.536230+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a26c0c1c-8de0-438c-a3f6-3df10ea61574","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5de24915-8dec-49b6-8b16-f3170197c591","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ac054af3-d589-4565-9644-e6af75975c53","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3ec4f045-c714-4a22-bd1a-e07330b45ab3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"cddb9867-8e30-4951-9b13-3d1e78b9c45a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dc6d9911-785d-432f-8b53-56a1ccab302f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"77c88437-b587-496e-8b4d-1824ea8b5305","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ce963d96-5a8e-44c3-99de-c0dbebe234ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:42:48.750 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:48.766+0000 7ffbb86f4640 10 librados: watch_flush enter 2024-03-29T10:42:48.750 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:48.766+0000 7ffbb86f4640 10 librados: watch_flush exit 2024-03-29T10:42:48.750 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:48.766+0000 7ffbb86f4640 20 client.4313.objecter shutdown clearing up homeless session... 2024-03-29T10:42:48.750 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:48.766+0000 7ffbb86f4640 10 client.4313.objecter successfully canceled tick 2024-03-29T10:42:48.750 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:48.766+0000 7ffbb86f4640 1 -- 172.21.15.88:0/1022522417 >> v1:172.21.15.88:6816/1752861349 conn(0x7ffb9406dbb0 legacy=0x7ffb94070070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:48.750 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:48.766+0000 7ffbb86f4640 10 monclient: shutdown 2024-03-29T10:42:48.750 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:48.766+0000 7ffbb86f4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:42:48.751 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:48.766+0000 7ffbb86f4640 1 -- 172.21.15.88:0/1022522417 >> v1:172.21.15.88:6790/0 conn(0x7ffbb00a9750 legacy=0x7ffbb01db7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:48.751 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:48.767+0000 7ffbb86f4640 1 -- 172.21.15.88:0/1022522417 shutdown_connections 2024-03-29T10:42:48.751 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:48.767+0000 7ffbb86f4640 1 -- 172.21.15.88:0/1022522417 >> 172.21.15.88:0/1022522417 conn(0x7ffbb00b3a00 msgr2=0x7ffbb00b3de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:42:48.751 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:48.767+0000 7ffbb86f4640 1 -- 172.21.15.88:0/1022522417 shutdown_connections 2024-03-29T10:42:48.751 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:48.767+0000 7ffbb86f4640 1 -- 172.21.15.88:0/1022522417 wait complete. 2024-03-29T10:42:48.751 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:48.767+0000 7ffbb86f4640 1 librados: shutdown 2024-03-29T10:42:48.767 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:42:55.574 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:42:55.724 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:55.740+0000 7f76ac69d640 10 monclient: get_monmap_and_config 2024-03-29T10:42:55.724 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:55.740+0000 7f76ac69d640 10 monclient: build_initial_monmap 2024-03-29T10:42:55.724 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:55.740+0000 7f76ac69d640 10 monclient: monmap: 2024-03-29T10:42:55.725 INFO:teuthology.misc.health.smithi088.stderr:epoch 0 2024-03-29T10:42:55.725 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:42:55.725 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:42:55.741985+0000 2024-03-29T10:42:55.725 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:42:55.741985+0000 2024-03-29T10:42:55.725 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 0 (unknown) 2024-03-29T10:42:55.725 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:42:55.725 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-03-29T10:42:55.725 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-29T10:42:55.725 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-b 2024-03-29T10:42:55.725 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:42:55.726 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:55.742+0000 7f76ac69d640 1 Processor -- start 2024-03-29T10:42:55.729 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:55.742+0000 7f76ac69d640 1 -- start start 2024-03-29T10:42:55.729 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:55.742+0000 7f76ac69d640 10 monclient: init 2024-03-29T10:42:55.729 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:55.743+0000 7f76ac69d640 10 monclient: _reopen_session rank -1 2024-03-29T10:42:55.729 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:55.743+0000 7f76ac69d640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:42:55.729 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:55.743+0000 7f76ac69d640 10 monclient(hunting): picked mon.noname-c con 0x7f76a40a9ac0 addr v1:172.21.15.196:6789/0 2024-03-29T10:42:55.729 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:55.743+0000 7f76ac69d640 10 monclient(hunting): picked mon.noname-a con 0x7f76a413ee90 addr v1:172.21.15.88:6789/0 2024-03-29T10:42:55.729 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:55.743+0000 7f76ac69d640 10 monclient(hunting): picked mon.noname-b con 0x7f76a40af900 addr v1:172.21.15.88:6790/0 2024-03-29T10:42:55.729 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:55.743+0000 7f76ac69d640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76a40afd10 con 0x7f76a413ee90 2024-03-29T10:42:55.729 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:55.743+0000 7f76ac69d640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76a40af6f0 con 0x7f76a40a9ac0 2024-03-29T10:42:55.729 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:55.743+0000 7f76ac69d640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76a40aac80 con 0x7f76a40af900 2024-03-29T10:42:55.729 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:55.743+0000 7f76ac69d640 10 monclient(hunting): _renew_subs 2024-03-29T10:42:55.729 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:55.743+0000 7f76ac69d640 10 monclient(hunting): authenticate will time out at 897.979080s 2024-03-29T10:42:55.730 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:55.747+0000 7f76aac3a640 1 --1- >> v1:172.21.15.88:6790/0 conn(0x7f76a40af900 0x7f76a4149b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6790/0 says I am v1:172.21.15.88:54383/0 (socket says 172.21.15.88:54383) 2024-03-29T10:42:55.730 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:55.747+0000 7f76aac3a640 1 -- 172.21.15.88:0/4005317771 learned_addr learned my addr 172.21.15.88:0/4005317771 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-29T10:42:55.741 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:55.753+0000 7f76a9437640 1 -- 172.21.15.88:0/4005317771 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3746172954 0 0) 0x7f76a40aac80 con 0x7f76a40af900 2024-03-29T10:42:55.741 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:55.753+0000 7f76a9437640 10 monclient(hunting): _negotiate 2024-03-29T10:42:55.741 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:55.753+0000 7f76a9437640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:42:55.741 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:55.753+0000 7f76a9437640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:42:55.741 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:55.753+0000 7f76a9437640 10 monclient(hunting): my global_id is 4316 2024-03-29T10:42:55.741 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:55.753+0000 7f76a9437640 1 -- 172.21.15.88:0/4005317771 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7690003520 con 0x7f76a40af900 2024-03-29T10:42:55.741 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:55.753+0000 7f76a9437640 1 -- 172.21.15.88:0/4005317771 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4025039175 0 0) 0x7f76a40af6f0 con 0x7f76a40a9ac0 2024-03-29T10:42:55.741 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:55.753+0000 7f76a9437640 10 monclient(hunting): _negotiate 2024-03-29T10:42:55.741 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:55.753+0000 7f76a9437640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:42:55.742 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:55.753+0000 7f76a9437640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:42:55.742 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:55.753+0000 7f76a9437640 10 monclient(hunting): my global_id is 4297 2024-03-29T10:42:55.742 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:55.753+0000 7f76a9437640 1 -- 172.21.15.88:0/4005317771 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f76a40aac80 con 0x7f76a40a9ac0 2024-03-29T10:42:55.742 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:55.753+0000 7f76a9437640 1 -- 172.21.15.88:0/4005317771 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1738936881 0 0) 0x7f76a40aac80 con 0x7f76a40a9ac0 2024-03-29T10:42:55.742 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:55.753+0000 7f76a9437640 1 -- 172.21.15.88:0/4005317771 >> v1:172.21.15.88:6790/0 conn(0x7f76a40af900 legacy=0x7f76a4149b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:55.742 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:55.753+0000 7f76a9437640 1 -- 172.21.15.88:0/4005317771 >> v1:172.21.15.88:6789/0 conn(0x7f76a413ee90 legacy=0x7f76a413b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:42:55.742 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:55.753+0000 7f76a9437640 10 monclient: _finish_hunting 0 2024-03-29T10:42:55.742 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:55.753+0000 7f76a9437640 1 monclient: found mon.noname-c 2024-03-29T10:42:55.742 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:55.753+0000 7f76a9437640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:42:55.742 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:55.753+0000 7f76a9437640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.196:6789/0 2024-03-29T10:42:55.742 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:55.753+0000 7f76a9437640 1 -- 172.21.15.88:0/4005317771 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76a40aaf60 con 0x7f76a40a9ac0 2024-03-29T10:42:55.742 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:55.753+0000 7f76a9437640 10 monclient: _finish_auth 0 2024-03-29T10:42:55.742 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:55.753+0000 7f76a9437640 10 monclient: _check_auth_tickets 2024-03-29T10:42:55.742 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:55.753+0000 7f76a9437640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:42:55.743 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:55.753+0000 7f76a9437640 1 -- 172.21.15.88:0/4005317771 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f769c001970 con 0x7f76a40a9ac0 2024-03-29T10:42:55.743 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:55.753+0000 7f76a9437640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:55.743 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:55.753+0000 7f76a9437640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:42:55.743 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:55.753+0000 7f76a9437640 10 monclient: dump: 2024-03-29T10:42:55.743 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:42:55.743 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:42:55.743 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:42:55.743 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:42:55.743 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:55.743 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:42:55.743 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:42:55.743 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:42:55.743 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:42:55.743 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:42:55.744 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:55.753+0000 7f76ac69d640 5 monclient: authenticate success, global_id 4297 2024-03-29T10:42:55.744 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:55.753+0000 7f76ac69d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:42:55.744 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:55.754+0000 7f76a9437640 1 -- 172.21.15.88:0/4005317771 <== mon.1 v1:172.21.15.196:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f769c001ad0 con 0x7f76a40a9ac0 2024-03-29T10:42:55.744 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:55.754+0000 7f76a9437640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:42:55.744 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:55.754+0000 7f76a9437640 1 -- 172.21.15.88:0/4005317771 <== mon.1 v1:172.21.15.196:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f769c001f20 con 0x7f76a40a9ac0 2024-03-29T10:42:55.744 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:55.754+0000 7f76a9437640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:55.744 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:55.754+0000 7f76a9437640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:42:55.744 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:55.754+0000 7f76a9437640 10 monclient: dump: 2024-03-29T10:42:55.744 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:42:55.744 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:42:55.744 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:42:55.744 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:42:55.744 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:55.744 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:42:55.745 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:42:55.745 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:42:55.745 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:42:55.745 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:42:55.745 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:55.754+0000 7f76ac69d640 10 monclient: get_monmap_and_config success 2024-03-29T10:42:55.745 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:55.754+0000 7f76ac69d640 4 set_mon_vals no callback set 2024-03-29T10:42:55.745 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:55.755+0000 7f76ac69d640 10 monclient: shutdown 2024-03-29T10:42:55.745 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:55.755+0000 7f76ac69d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:42:55.745 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:55.755+0000 7f76ac69d640 1 -- 172.21.15.88:0/4005317771 >> v1:172.21.15.196:6789/0 conn(0x7f76a40a9ac0 legacy=0x7f76a40adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:55.745 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:55.755+0000 7f76ac69d640 1 -- 172.21.15.88:0/4005317771 shutdown_connections 2024-03-29T10:42:55.745 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:55.755+0000 7f76ac69d640 1 -- 172.21.15.88:0/4005317771 >> 172.21.15.88:0/4005317771 conn(0x7f76a40b35e0 msgr2=0x7f76a40b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:42:55.745 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:55.756+0000 7f76ac69d640 1 -- 172.21.15.88:0/4005317771 shutdown_connections 2024-03-29T10:42:55.745 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:55.756+0000 7f76ac69d640 1 -- 172.21.15.88:0/4005317771 wait complete. 2024-03-29T10:42:55.746 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:55.756+0000 7f76ac69d640 10 monclient: build_initial_monmap 2024-03-29T10:42:55.746 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:55.756+0000 7f76ac69d640 10 monclient: monmap: 2024-03-29T10:42:55.746 INFO:teuthology.misc.health.smithi088.stderr:epoch 0 2024-03-29T10:42:55.746 INFO:teuthology.misc.health.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:42:55.746 INFO:teuthology.misc.health.smithi088.stderr:last_changed 0.000000 2024-03-29T10:42:55.746 INFO:teuthology.misc.health.smithi088.stderr:created 0.000000 2024-03-29T10:42:55.746 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 0 (unknown) 2024-03-29T10:42:55.746 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:42:55.746 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-03-29T10:42:55.746 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-29T10:42:55.746 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-c 2024-03-29T10:42:55.746 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:42:55.746 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:55.757+0000 7f76ac69d640 1 librados: starting msgr at 2024-03-29T10:42:55.747 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:55.757+0000 7f76ac69d640 1 librados: starting objecter 2024-03-29T10:42:55.747 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:55.757+0000 7f76ac69d640 1 Processor -- start 2024-03-29T10:42:55.748 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:55.757+0000 7f76ac69d640 1 -- start start 2024-03-29T10:42:55.748 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:55.757+0000 7f76ac69d640 1 librados: setting wanted keys 2024-03-29T10:42:55.748 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:55.757+0000 7f76ac69d640 1 librados: calling monclient init 2024-03-29T10:42:55.748 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:55.757+0000 7f76ac69d640 10 monclient: init 2024-03-29T10:42:55.749 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:55.760+0000 7f76ac69d640 10 monclient: _reopen_session rank -1 2024-03-29T10:42:55.749 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:55.760+0000 7f76ac69d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:42:55.749 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:55.760+0000 7f76ac69d640 10 monclient(hunting): picked mon.noname-c con 0x7f76a40a9ac0 addr v1:172.21.15.88:6790/0 2024-03-29T10:42:55.749 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:55.760+0000 7f76ac69d640 10 monclient(hunting): picked mon.noname-b con 0x7f76a40af900 addr v1:172.21.15.196:6789/0 2024-03-29T10:42:55.749 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:55.760+0000 7f76ac69d640 10 monclient(hunting): picked mon.noname-a con 0x7f76a413ee90 addr v1:172.21.15.88:6789/0 2024-03-29T10:42:55.749 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:55.760+0000 7f76ac69d640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76a41e7600 con 0x7f76a413ee90 2024-03-29T10:42:55.749 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:55.760+0000 7f76ac69d640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76a412d2e0 con 0x7f76a40af900 2024-03-29T10:42:55.749 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:55.760+0000 7f76ac69d640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76a4071590 con 0x7f76a40a9ac0 2024-03-29T10:42:55.749 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:55.760+0000 7f76ac69d640 10 monclient(hunting): _renew_subs 2024-03-29T10:42:55.749 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:55.760+0000 7f76ac69d640 10 monclient(hunting): authenticate will time out at 897.996338s 2024-03-29T10:42:55.754 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:55.765+0000 7f76aa439640 1 --1- >> v1:172.21.15.88:6790/0 conn(0x7f76a40a9ac0 0x7f76a4072070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6790/0 says I am v1:172.21.15.88:54477/0 (socket says 172.21.15.88:54477) 2024-03-29T10:42:55.754 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:55.765+0000 7f76aa439640 1 -- 172.21.15.88:0/3876432904 learned_addr learned my addr 172.21.15.88:0/3876432904 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-29T10:42:55.754 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:55.765+0000 7f769affd640 10 client.?.objecter ms_handle_connect 0x7f76a40a9ac0 2024-03-29T10:42:55.754 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:55.765+0000 7f769affd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:42:55.754 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:55.765+0000 7f769affd640 1 -- 172.21.15.88:0/3876432904 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3724430439 0 0) 0x7f76a4071590 con 0x7f76a40a9ac0 2024-03-29T10:42:55.754 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:55.765+0000 7f769affd640 10 monclient(hunting): _negotiate 2024-03-29T10:42:55.754 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:55.765+0000 7f769affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:42:55.754 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:55.765+0000 7f769affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:42:55.754 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:55.765+0000 7f769affd640 10 monclient(hunting): my global_id is 4319 2024-03-29T10:42:55.754 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:55.765+0000 7f769affd640 1 -- 172.21.15.88:0/3876432904 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7688004eb0 con 0x7f76a40a9ac0 2024-03-29T10:42:55.754 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:55.765+0000 7f769affd640 1 -- 172.21.15.88:0/3876432904 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 764900416 0 0) 0x7f7688004eb0 con 0x7f76a40a9ac0 2024-03-29T10:42:55.754 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:55.765+0000 7f769affd640 1 -- 172.21.15.88:0/3876432904 --> v1:172.21.15.88:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f76a4071590 con 0x7f76a40a9ac0 2024-03-29T10:42:55.755 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:55.765+0000 7f769affd640 1 -- 172.21.15.88:0/3876432904 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f769c001f20 con 0x7f76a40a9ac0 2024-03-29T10:42:55.755 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:55.765+0000 7f769affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:55.755 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:55.765+0000 7f769affd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:42:55.755 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:55.765+0000 7f769affd640 10 monclient(hunting): dump: 2024-03-29T10:42:55.755 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:42:55.755 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:42:55.755 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:42:55.755 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:42:55.755 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:55.755 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:42:55.755 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:42:55.755 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:42:55.755 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:42:55.756 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:42:55.756 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:55.765+0000 7f769affd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:42:55.756 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:55.766+0000 7f769affd640 1 -- 172.21.15.88:0/3876432904 <== mon.2 v1:172.21.15.88:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4120463305 0 0) 0x7f76a4071590 con 0x7f76a40a9ac0 2024-03-29T10:42:55.756 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:55.766+0000 7f769affd640 1 -- 172.21.15.88:0/3876432904 >> v1:172.21.15.196:6789/0 conn(0x7f76a40af900 legacy=0x7f76a41e7a90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:42:55.756 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:55.768+0000 7f769affd640 1 -- 172.21.15.88:0/3876432904 >> v1:172.21.15.88:6789/0 conn(0x7f76a413ee90 legacy=0x7f76a41dabb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:42:55.756 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:55.768+0000 7f769affd640 10 monclient: _finish_hunting 0 2024-03-29T10:42:55.756 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:55.768+0000 7f769affd640 1 monclient: found mon.c 2024-03-29T10:42:55.756 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:55.768+0000 7f769affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:42:55.756 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:55.768+0000 7f769affd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-03-29T10:42:55.756 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:55.768+0000 7f769affd640 1 -- 172.21.15.88:0/3876432904 --> v1:172.21.15.88:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76a406ec40 con 0x7f76a40a9ac0 2024-03-29T10:42:55.756 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:55.768+0000 7f769affd640 10 monclient: _finish_auth 0 2024-03-29T10:42:55.756 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:55.768+0000 7f769affd640 10 monclient: _check_auth_tickets 2024-03-29T10:42:55.756 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:55.768+0000 7f769affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:42:55.756 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:55.768+0000 7f76ac69d640 5 monclient: authenticate success, global_id 4319 2024-03-29T10:42:55.757 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:55.768+0000 7f76ac69d640 10 monclient: _renew_subs 2024-03-29T10:42:55.757 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:55.768+0000 7f76ac69d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-03-29T10:42:55.757 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:55.768+0000 7f76ac69d640 1 -- 172.21.15.88:0/3876432904 --> v1:172.21.15.88:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76a41db2c0 con 0x7f76a40a9ac0 2024-03-29T10:42:55.757 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:55.768+0000 7f76ac69d640 10 client.4319.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:42:55.757 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:55.768+0000 7f76ac69d640 10 monclient: _renew_subs 2024-03-29T10:42:55.757 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:55.768+0000 7f76ac69d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-03-29T10:42:55.757 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:55.768+0000 7f76ac69d640 1 -- 172.21.15.88:0/3876432904 --> v1:172.21.15.88:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f76a41e8380 con 0x7f76a40a9ac0 2024-03-29T10:42:55.757 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:55.768+0000 7f76ac69d640 1 librados: init done 2024-03-29T10:42:55.757 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:55.769+0000 7f769affd640 1 -- 172.21.15.88:0/3876432904 <== mon.2 v1:172.21.15.88:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f769c003c20 con 0x7f76a40a9ac0 2024-03-29T10:42:55.757 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:55.769+0000 7f769affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:42:55.757 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:55.769+0000 7f769affd640 1 -- 172.21.15.88:0/3876432904 <== mon.2 v1:172.21.15.88:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f769c001530 con 0x7f76a40a9ac0 2024-03-29T10:42:55.757 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:55.769+0000 7f769affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:55.757 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:55.769+0000 7f769affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:42:55.757 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:55.769+0000 7f769affd640 10 monclient: dump: 2024-03-29T10:42:55.758 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:42:55.758 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:42:55.758 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:42:55.758 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:42:55.758 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:55.758 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:42:55.758 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:42:55.758 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:42:55.758 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:42:55.758 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:42:55.758 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:55.770+0000 7f76a8c36640 4 set_mon_vals no callback set 2024-03-29T10:42:55.758 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:55.770+0000 7f76ac69d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:42:55.759 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:55.771+0000 7f769affd640 1 -- 172.21.15.88:0/3876432904 <== mon.2 v1:172.21.15.88:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4235629220 0 0) 0x7f769c0040f0 con 0x7f76a40a9ac0 2024-03-29T10:42:55.759 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:55.771+0000 7f76ac69d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:42:55.760 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:55.771+0000 7f76ac69d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-03-29T10:42:55.760 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:55.771+0000 7f76ac69d640 1 -- 172.21.15.88:0/3876432904 --> v1:172.21.15.88:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76a413bd00 con 0x7f76a40a9ac0 2024-03-29T10:42:55.760 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:55.772+0000 7f769affd640 1 -- 172.21.15.88:0/3876432904 <== mon.2 v1:172.21.15.88:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4257673582 0 0) 0x7f769c085060 con 0x7f76a40a9ac0 2024-03-29T10:42:55.760 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:55.772+0000 7f769affd640 10 client.4319.objecter ms_dispatch 0x7f76a4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:42:55.760 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:55.772+0000 7f769affd640 3 client.4319.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:42:55.760 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:55.772+0000 7f769affd640 3 client.4319.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:42:55.760 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:55.774+0000 7f769affd640 20 client.4319.objecter dump_active .. 0 homeless 2024-03-29T10:42:55.760 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:55.774+0000 7f76a9c38640 10 monclient: get_auth_request con 0x7f768806db80 auth_method 0 2024-03-29T10:42:55.760 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:55.774+0000 7f769affd640 10 client.4319.objecter ms_handle_connect 0x7f768806db80 2024-03-29T10:42:55.768 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:55.784+0000 7f769affd640 1 -- 172.21.15.88:0/3876432904 <== mon.2 v1:172.21.15.88:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f76a413bd00 con 0x7f76a40a9ac0 2024-03-29T10:42:55.768 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:55.784+0000 7f769affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:42:55.768 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:55.784+0000 7f769affd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:42:56.095 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:56.110+0000 7f76ac69d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:42:56.095 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:56.110+0000 7f76ac69d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:42:56.095 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:56.110+0000 7f76ac69d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-03-29T10:42:56.095 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:56.110+0000 7f76ac69d640 1 -- 172.21.15.88:0/3876432904 --> v1:172.21.15.88:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f76a413bd00 con 0x7f76a40a9ac0 2024-03-29T10:42:56.095 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:56.111+0000 7f769affd640 1 -- 172.21.15.88:0/3876432904 <== mon.2 v1:172.21.15.88:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 344506374) 0x7f76a413bd00 con 0x7f76a40a9ac0 2024-03-29T10:42:56.095 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:56.111+0000 7f769affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:42:56.095 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:56.112+0000 7f769affd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:42:56.096 INFO:teuthology.misc.health.smithi088.stdout: 2024-03-29T10:42:56.096 INFO:teuthology.misc.health.smithi088.stdout:{"epoch":10,"fsid":"869f5188-f92b-42dd-a46d-8c646e0388a9","created":"2024-03-29T10:40:40.285087+0000","modified":"2024-03-29T10:40:42.536230+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:40:42.536230+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a26c0c1c-8de0-438c-a3f6-3df10ea61574","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5de24915-8dec-49b6-8b16-f3170197c591","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ac054af3-d589-4565-9644-e6af75975c53","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3ec4f045-c714-4a22-bd1a-e07330b45ab3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"cddb9867-8e30-4951-9b13-3d1e78b9c45a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dc6d9911-785d-432f-8b53-56a1ccab302f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"77c88437-b587-496e-8b4d-1824ea8b5305","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ce963d96-5a8e-44c3-99de-c0dbebe234ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:42:56.099 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:56.115+0000 7f76ac69d640 10 librados: watch_flush enter 2024-03-29T10:42:56.099 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:56.115+0000 7f76ac69d640 10 librados: watch_flush exit 2024-03-29T10:42:56.099 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:56.115+0000 7f76ac69d640 20 client.4319.objecter shutdown clearing up homeless session... 2024-03-29T10:42:56.099 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:56.115+0000 7f76ac69d640 10 client.4319.objecter successfully canceled tick 2024-03-29T10:42:56.099 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:56.115+0000 7f76ac69d640 1 -- 172.21.15.88:0/3876432904 >> v1:172.21.15.88:6816/1752861349 conn(0x7f768806db80 legacy=0x7f7688070040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:56.099 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:56.115+0000 7f76ac69d640 10 monclient: shutdown 2024-03-29T10:42:56.099 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:56.115+0000 7f76ac69d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:42:56.099 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:56.115+0000 7f76ac69d640 1 -- 172.21.15.88:0/3876432904 >> v1:172.21.15.88:6790/0 conn(0x7f76a40a9ac0 legacy=0x7f76a4072070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:56.100 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:56.117+0000 7f76ac69d640 1 -- 172.21.15.88:0/3876432904 shutdown_connections 2024-03-29T10:42:56.100 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:56.117+0000 7f76ac69d640 1 -- 172.21.15.88:0/3876432904 >> 172.21.15.88:0/3876432904 conn(0x7f76a40b35e0 msgr2=0x7f76a413b000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:42:56.103 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:56.120+0000 7f76ac69d640 1 -- 172.21.15.88:0/3876432904 shutdown_connections 2024-03-29T10:42:56.103 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:56.120+0000 7f76ac69d640 1 -- 172.21.15.88:0/3876432904 wait complete. 2024-03-29T10:42:56.103 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:42:56.120+0000 7f76ac69d640 1 librados: shutdown 2024-03-29T10:42:56.122 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:43:02.928 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:43:03.087 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:03.103+0000 7f183759e640 10 monclient: get_monmap_and_config 2024-03-29T10:43:03.087 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:03.103+0000 7f183759e640 10 monclient: build_initial_monmap 2024-03-29T10:43:03.087 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:03.103+0000 7f183759e640 10 monclient: monmap: 2024-03-29T10:43:03.087 INFO:teuthology.misc.health.smithi088.stderr:epoch 0 2024-03-29T10:43:03.087 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:43:03.087 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:43:03.105007+0000 2024-03-29T10:43:03.087 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:43:03.105007+0000 2024-03-29T10:43:03.087 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 0 (unknown) 2024-03-29T10:43:03.087 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:43:03.088 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-03-29T10:43:03.088 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-29T10:43:03.088 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-b 2024-03-29T10:43:03.088 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:43:03.088 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:03.104+0000 7f183759e640 1 Processor -- start 2024-03-29T10:43:03.088 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:03.105+0000 7f183759e640 1 -- start start 2024-03-29T10:43:03.089 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:03.105+0000 7f183759e640 10 monclient: init 2024-03-29T10:43:03.089 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:03.106+0000 7f183759e640 10 monclient: _reopen_session rank -1 2024-03-29T10:43:03.089 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:03.106+0000 7f183759e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:43:03.089 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:03.106+0000 7f183759e640 10 monclient(hunting): picked mon.noname-a con 0x7f18300389e0 addr v1:172.21.15.88:6789/0 2024-03-29T10:43:03.089 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:03.106+0000 7f183759e640 10 monclient(hunting): picked mon.noname-c con 0x7f1830038dc0 addr v1:172.21.15.196:6789/0 2024-03-29T10:43:03.089 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:03.106+0000 7f183759e640 10 monclient(hunting): picked mon.noname-b con 0x7f1830030660 addr v1:172.21.15.88:6790/0 2024-03-29T10:43:03.089 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:03.106+0000 7f183759e640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18300c54c0 con 0x7f18300389e0 2024-03-29T10:43:03.089 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:03.106+0000 7f183759e640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1830036430 con 0x7f1830038dc0 2024-03-29T10:43:03.090 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:03.106+0000 7f183759e640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1830039840 con 0x7f1830030660 2024-03-29T10:43:03.090 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:03.106+0000 7f183759e640 10 monclient(hunting): _renew_subs 2024-03-29T10:43:03.090 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:03.106+0000 7f183759e640 10 monclient(hunting): authenticate will time out at 905.342087s 2024-03-29T10:43:03.090 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:03.106+0000 7f183659c640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f18300389e0 0x7f1830033520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:42878/0 (socket says 172.21.15.88:42878) 2024-03-29T10:43:03.090 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:03.106+0000 7f183659c640 1 -- 172.21.15.88:0/2792905643 learned_addr learned my addr 172.21.15.88:0/2792905643 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-29T10:43:03.094 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:03.107+0000 7f183559a640 1 -- 172.21.15.88:0/2792905643 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3851427512 0 0) 0x7f1830036430 con 0x7f1830038dc0 2024-03-29T10:43:03.094 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:03.107+0000 7f183559a640 10 monclient(hunting): _negotiate 2024-03-29T10:43:03.094 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:03.107+0000 7f183559a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:43:03.094 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:03.107+0000 7f183559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:43:03.094 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:03.107+0000 7f183559a640 10 monclient(hunting): my global_id is 4300 2024-03-29T10:43:03.094 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:03.107+0000 7f183559a640 1 -- 172.21.15.88:0/2792905643 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1820003520 con 0x7f1830038dc0 2024-03-29T10:43:03.094 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:03.107+0000 7f183559a640 1 -- 172.21.15.88:0/2792905643 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2119308713 0 0) 0x7f1830039840 con 0x7f1830030660 2024-03-29T10:43:03.094 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:03.107+0000 7f183559a640 10 monclient(hunting): _negotiate 2024-03-29T10:43:03.094 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:03.107+0000 7f183559a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:43:03.095 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:03.107+0000 7f183559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:43:03.095 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:03.107+0000 7f183559a640 10 monclient(hunting): my global_id is 4322 2024-03-29T10:43:03.095 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:03.107+0000 7f183559a640 1 -- 172.21.15.88:0/2792905643 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1830036430 con 0x7f1830030660 2024-03-29T10:43:03.095 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:03.107+0000 7f183559a640 1 -- 172.21.15.88:0/2792905643 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3323623110 0 0) 0x7f18300c54c0 con 0x7f18300389e0 2024-03-29T10:43:03.095 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:03.107+0000 7f183559a640 10 monclient(hunting): _negotiate 2024-03-29T10:43:03.095 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:03.107+0000 7f183559a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:43:03.095 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:03.107+0000 7f183559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:43:03.095 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:03.107+0000 7f183559a640 10 monclient(hunting): my global_id is 4308 2024-03-29T10:43:03.095 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:03.107+0000 7f183559a640 1 -- 172.21.15.88:0/2792905643 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1830039840 con 0x7f18300389e0 2024-03-29T10:43:03.095 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:03.108+0000 7f183559a640 1 -- 172.21.15.88:0/2792905643 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3706482237 0 0) 0x7f1820003520 con 0x7f1830038dc0 2024-03-29T10:43:03.095 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:03.108+0000 7f183559a640 1 -- 172.21.15.88:0/2792905643 >> v1:172.21.15.88:6790/0 conn(0x7f1830030660 legacy=0x7f18300d3170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:03.095 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:03.108+0000 7f183559a640 1 -- 172.21.15.88:0/2792905643 >> v1:172.21.15.88:6789/0 conn(0x7f18300389e0 legacy=0x7f1830033520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:03.095 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:03.108+0000 7f183559a640 10 monclient: _finish_hunting 0 2024-03-29T10:43:03.095 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:03.108+0000 7f183559a640 1 monclient: found mon.noname-c 2024-03-29T10:43:03.096 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:03.108+0000 7f183559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:43:03.096 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:03.108+0000 7f183559a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.196:6789/0 2024-03-29T10:43:03.096 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:03.108+0000 7f183559a640 1 -- 172.21.15.88:0/2792905643 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1830035690 con 0x7f1830038dc0 2024-03-29T10:43:03.096 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:03.108+0000 7f183559a640 10 monclient: _finish_auth 0 2024-03-29T10:43:03.096 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:03.108+0000 7f183559a640 10 monclient: _check_auth_tickets 2024-03-29T10:43:03.096 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:03.108+0000 7f183559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:43:03.096 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:03.108+0000 7f183759e640 5 monclient: authenticate success, global_id 4300 2024-03-29T10:43:03.096 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:03.108+0000 7f183759e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:43:03.096 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:03.108+0000 7f183559a640 1 -- 172.21.15.88:0/2792905643 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f182c001690 con 0x7f1830038dc0 2024-03-29T10:43:03.096 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:03.108+0000 7f183559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:43:03.096 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:03.108+0000 7f183559a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:43:03.096 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:03.108+0000 7f183559a640 10 monclient: dump: 2024-03-29T10:43:03.096 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:43:03.096 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:43:03.097 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:43:03.097 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:43:03.097 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:43:03.097 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:43:03.097 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:43:03.097 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:43:03.097 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:43:03.097 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:43:03.097 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:03.108+0000 7f183759e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:43:03.097 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:03.108+0000 7f183559a640 1 -- 172.21.15.88:0/2792905643 <== mon.1 v1:172.21.15.196:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f182c0018b0 con 0x7f1830038dc0 2024-03-29T10:43:03.097 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:03.108+0000 7f183559a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:43:03.097 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:03.108+0000 7f183759e640 10 monclient: get_monmap_and_config success 2024-03-29T10:43:03.097 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:03.108+0000 7f183759e640 4 set_mon_vals no callback set 2024-03-29T10:43:03.097 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:03.108+0000 7f183559a640 1 -- 172.21.15.88:0/2792905643 <== mon.1 v1:172.21.15.196:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f182c001c40 con 0x7f1830038dc0 2024-03-29T10:43:03.098 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:03.108+0000 7f183559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:43:03.098 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:03.108+0000 7f183559a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:43:03.098 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:03.108+0000 7f183559a640 10 monclient: dump: 2024-03-29T10:43:03.098 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:43:03.098 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:43:03.098 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:43:03.098 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:43:03.098 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:43:03.098 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:43:03.098 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:43:03.098 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:43:03.098 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:43:03.098 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:43:03.099 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:03.109+0000 7f183759e640 10 monclient: shutdown 2024-03-29T10:43:03.099 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:03.109+0000 7f183759e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:43:03.099 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:03.109+0000 7f183759e640 1 -- 172.21.15.88:0/2792905643 >> v1:172.21.15.196:6789/0 conn(0x7f1830038dc0 legacy=0x7f18300cadd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:03.099 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:03.109+0000 7f183759e640 1 -- 172.21.15.88:0/2792905643 shutdown_connections 2024-03-29T10:43:03.099 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:03.109+0000 7f183759e640 1 -- 172.21.15.88:0/2792905643 >> 172.21.15.88:0/2792905643 conn(0x7f1830005c00 msgr2=0x7f1830006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:43:03.099 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:03.109+0000 7f183759e640 1 -- 172.21.15.88:0/2792905643 shutdown_connections 2024-03-29T10:43:03.099 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:03.109+0000 7f183759e640 1 -- 172.21.15.88:0/2792905643 wait complete. 2024-03-29T10:43:03.099 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:03.110+0000 7f183759e640 10 monclient: build_initial_monmap 2024-03-29T10:43:03.099 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:03.110+0000 7f183759e640 10 monclient: monmap: 2024-03-29T10:43:03.099 INFO:teuthology.misc.health.smithi088.stderr:epoch 0 2024-03-29T10:43:03.100 INFO:teuthology.misc.health.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:43:03.100 INFO:teuthology.misc.health.smithi088.stderr:last_changed 0.000000 2024-03-29T10:43:03.100 INFO:teuthology.misc.health.smithi088.stderr:created 0.000000 2024-03-29T10:43:03.100 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 0 (unknown) 2024-03-29T10:43:03.100 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:43:03.100 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-03-29T10:43:03.100 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-29T10:43:03.100 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-c 2024-03-29T10:43:03.100 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:43:03.100 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:03.111+0000 7f183759e640 1 librados: starting msgr at 2024-03-29T10:43:03.100 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:03.111+0000 7f183759e640 1 librados: starting objecter 2024-03-29T10:43:03.100 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:03.111+0000 7f183759e640 1 Processor -- start 2024-03-29T10:43:03.100 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:03.111+0000 7f183759e640 1 -- start start 2024-03-29T10:43:03.100 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:03.111+0000 7f183759e640 1 librados: setting wanted keys 2024-03-29T10:43:03.100 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:03.111+0000 7f183759e640 1 librados: calling monclient init 2024-03-29T10:43:03.101 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:03.111+0000 7f183759e640 10 monclient: init 2024-03-29T10:43:03.103 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:03.111+0000 7f183759e640 10 monclient: _reopen_session rank -1 2024-03-29T10:43:03.103 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:03.111+0000 7f183759e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:43:03.103 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:03.111+0000 7f183759e640 10 monclient(hunting): picked mon.noname-a con 0x7f1830030660 addr v1:172.21.15.88:6789/0 2024-03-29T10:43:03.103 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:03.111+0000 7f183759e640 10 monclient(hunting): picked mon.noname-b con 0x7f18300389e0 addr v1:172.21.15.196:6789/0 2024-03-29T10:43:03.103 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:03.111+0000 7f183759e640 10 monclient(hunting): picked mon.noname-c con 0x7f18300ca8f0 addr v1:172.21.15.88:6790/0 2024-03-29T10:43:03.103 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:03.111+0000 7f183759e640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18300a6160 con 0x7f1830030660 2024-03-29T10:43:03.103 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:03.111+0000 7f183759e640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18300cae10 con 0x7f18300389e0 2024-03-29T10:43:03.103 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:03.111+0000 7f183759e640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18300ca0f0 con 0x7f18300ca8f0 2024-03-29T10:43:03.103 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:03.111+0000 7f183759e640 10 monclient(hunting): _renew_subs 2024-03-29T10:43:03.103 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:03.111+0000 7f183759e640 10 monclient(hunting): authenticate will time out at 905.347704s 2024-03-29T10:43:03.104 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:03.112+0000 7f1835d9b640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f18300389e0 0x7f18301606e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.88:38784/0 (socket says 172.21.15.88:38784) 2024-03-29T10:43:03.104 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:03.112+0000 7f1835d9b640 1 -- 172.21.15.88:0/1944034523 learned_addr learned my addr 172.21.15.88:0/1944034523 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-29T10:43:03.104 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:03.112+0000 7f1826ffd640 10 client.?.objecter ms_handle_connect 0x7f18300389e0 2024-03-29T10:43:03.104 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:03.112+0000 7f1826ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:43:03.104 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:03.112+0000 7f1826ffd640 10 client.?.objecter ms_handle_connect 0x7f18300ca8f0 2024-03-29T10:43:03.104 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:03.112+0000 7f1826ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:43:03.104 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:03.112+0000 7f1826ffd640 1 -- 172.21.15.88:0/1944034523 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1506969774 0 0) 0x7f18300ca0f0 con 0x7f18300ca8f0 2024-03-29T10:43:03.104 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:03.112+0000 7f1826ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:43:03.104 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:03.112+0000 7f1826ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:43:03.104 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:03.112+0000 7f1826ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:43:03.104 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:03.112+0000 7f1826ffd640 10 monclient(hunting): my global_id is 4325 2024-03-29T10:43:03.104 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:03.112+0000 7f1826ffd640 1 -- 172.21.15.88:0/1944034523 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1818004eb0 con 0x7f18300ca8f0 2024-03-29T10:43:03.104 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:03.113+0000 7f1826ffd640 1 -- 172.21.15.88:0/1944034523 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3103967520 0 0) 0x7f1818004eb0 con 0x7f18300ca8f0 2024-03-29T10:43:03.104 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:03.113+0000 7f1826ffd640 1 -- 172.21.15.88:0/1944034523 --> v1:172.21.15.88:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f18300ca0f0 con 0x7f18300ca8f0 2024-03-29T10:43:03.105 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:03.113+0000 7f1826ffd640 1 -- 172.21.15.88:0/1944034523 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f183804fdd0 con 0x7f18300ca8f0 2024-03-29T10:43:03.105 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:03.113+0000 7f1826ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:43:03.105 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:03.113+0000 7f1826ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:43:03.105 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:03.113+0000 7f1826ffd640 10 monclient(hunting): dump: 2024-03-29T10:43:03.105 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:43:03.105 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:43:03.105 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:43:03.105 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:43:03.105 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:43:03.105 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:43:03.105 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:43:03.105 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:43:03.105 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:43:03.105 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:43:03.105 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:03.113+0000 7f1826ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:43:03.106 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:03.113+0000 7f1826ffd640 1 -- 172.21.15.88:0/1944034523 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4158837659 0 0) 0x7f18300cae10 con 0x7f18300389e0 2024-03-29T10:43:03.106 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:03.113+0000 7f1826ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:43:03.106 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:03.113+0000 7f1826ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:43:03.106 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:03.113+0000 7f1826ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:43:03.106 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:03.113+0000 7f1826ffd640 10 monclient(hunting): my global_id is 4303 2024-03-29T10:43:03.106 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:03.113+0000 7f1826ffd640 1 -- 172.21.15.88:0/1944034523 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1818004eb0 con 0x7f18300389e0 2024-03-29T10:43:03.106 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:03.113+0000 7f1826ffd640 1 -- 172.21.15.88:0/1944034523 <== mon.2 v1:172.21.15.88:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2684006919 0 0) 0x7f18300ca0f0 con 0x7f18300ca8f0 2024-03-29T10:43:03.106 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:03.113+0000 7f1826ffd640 1 -- 172.21.15.88:0/1944034523 >> v1:172.21.15.196:6789/0 conn(0x7f18300389e0 legacy=0x7f18301606e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:03.106 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:03.113+0000 7f1826ffd640 1 -- 172.21.15.88:0/1944034523 >> v1:172.21.15.88:6789/0 conn(0x7f1830030660 legacy=0x7f183015ffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:03.106 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:03.113+0000 7f1826ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:43:03.106 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:03.113+0000 7f1826ffd640 1 monclient: found mon.c 2024-03-29T10:43:03.106 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:03.113+0000 7f1826ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:43:03.106 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:03.113+0000 7f1826ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-03-29T10:43:03.107 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:03.113+0000 7f1826ffd640 1 -- 172.21.15.88:0/1944034523 --> v1:172.21.15.88:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18300c3b10 con 0x7f18300ca8f0 2024-03-29T10:43:03.107 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:03.113+0000 7f1826ffd640 10 monclient: _finish_auth 0 2024-03-29T10:43:03.107 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:03.113+0000 7f1826ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:43:03.107 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:03.113+0000 7f1826ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:43:03.107 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:03.113+0000 7f183759e640 5 monclient: authenticate success, global_id 4325 2024-03-29T10:43:03.107 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:03.113+0000 7f183759e640 10 monclient: _renew_subs 2024-03-29T10:43:03.107 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:03.113+0000 7f183759e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-03-29T10:43:03.107 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:03.113+0000 7f183759e640 1 -- 172.21.15.88:0/1944034523 --> v1:172.21.15.88:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18300a4f00 con 0x7f18300ca8f0 2024-03-29T10:43:03.107 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:03.113+0000 7f183759e640 10 client.4325.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:43:03.107 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:03.113+0000 7f183759e640 10 monclient: _renew_subs 2024-03-29T10:43:03.108 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:03.113+0000 7f183759e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-03-29T10:43:03.108 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:03.113+0000 7f183759e640 1 -- 172.21.15.88:0/1944034523 --> v1:172.21.15.88:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1830168a20 con 0x7f18300ca8f0 2024-03-29T10:43:03.108 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:03.113+0000 7f183759e640 1 librados: init done 2024-03-29T10:43:03.108 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:03.114+0000 7f1826ffd640 1 -- 172.21.15.88:0/1944034523 <== mon.2 v1:172.21.15.88:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f183804f400 con 0x7f18300ca8f0 2024-03-29T10:43:03.108 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:03.114+0000 7f1826ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:43:03.108 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:03.114+0000 7f1826ffd640 1 -- 172.21.15.88:0/1944034523 <== mon.2 v1:172.21.15.88:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f1838075710 con 0x7f18300ca8f0 2024-03-29T10:43:03.108 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:03.114+0000 7f1826ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:43:03.108 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:03.114+0000 7f1826ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:43:03.108 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:03.114+0000 7f1826ffd640 10 monclient: dump: 2024-03-29T10:43:03.108 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:43:03.108 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:43:03.108 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:43:03.108 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:43:03.108 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:43:03.108 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:43:03.109 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:43:03.109 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:43:03.109 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:43:03.109 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:43:03.109 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:03.114+0000 7f1834d99640 4 set_mon_vals no callback set 2024-03-29T10:43:03.109 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:03.115+0000 7f183759e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:43:03.109 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:03.115+0000 7f183759e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:43:03.109 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:03.115+0000 7f183759e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-03-29T10:43:03.109 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:03.115+0000 7f183759e640 1 -- 172.21.15.88:0/1944034523 --> v1:172.21.15.88:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f181c005080 con 0x7f18300ca8f0 2024-03-29T10:43:03.109 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:03.115+0000 7f1826ffd640 1 -- 172.21.15.88:0/1944034523 <== mon.2 v1:172.21.15.88:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4235629220 0 0) 0x7f1838067bf0 con 0x7f18300ca8f0 2024-03-29T10:43:03.109 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:03.116+0000 7f1826ffd640 1 -- 172.21.15.88:0/1944034523 <== mon.2 v1:172.21.15.88:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4257673582 0 0) 0x7f18380fe410 con 0x7f18300ca8f0 2024-03-29T10:43:03.109 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:03.116+0000 7f1826ffd640 10 client.4325.objecter ms_dispatch 0x7f1838000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:43:03.109 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:03.116+0000 7f1826ffd640 3 client.4325.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:43:03.109 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:03.116+0000 7f1826ffd640 3 client.4325.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:43:03.109 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:03.116+0000 7f1826ffd640 20 client.4325.objecter dump_active .. 0 homeless 2024-03-29T10:43:03.110 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:03.116+0000 7f183659c640 10 monclient: get_auth_request con 0x7f181806db00 auth_method 0 2024-03-29T10:43:03.110 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:03.117+0000 7f1826ffd640 10 client.4325.objecter ms_handle_connect 0x7f181806db00 2024-03-29T10:43:03.110 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:03.126+0000 7f1826ffd640 1 -- 172.21.15.88:0/1944034523 <== mon.2 v1:172.21.15.88:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f18380c9d50 con 0x7f18300ca8f0 2024-03-29T10:43:03.110 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:03.126+0000 7f1826ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:43:03.110 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:03.126+0000 7f1826ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:43:03.467 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:03.484+0000 7f183759e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:43:03.467 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:03.484+0000 7f183759e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:43:03.468 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:03.484+0000 7f183759e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-03-29T10:43:03.468 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:03.484+0000 7f183759e640 1 -- 172.21.15.88:0/1944034523 --> v1:172.21.15.88:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f181c005330 con 0x7f18300ca8f0 2024-03-29T10:43:03.468 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:03.485+0000 7f1826ffd640 1 -- 172.21.15.88:0/1944034523 <== mon.2 v1:172.21.15.88:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 344506374) 0x7f1838067f20 con 0x7f18300ca8f0 2024-03-29T10:43:03.468 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:03.485+0000 7f1826ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:43:03.468 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:03.485+0000 7f1826ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:43:03.469 INFO:teuthology.misc.health.smithi088.stdout: 2024-03-29T10:43:03.469 INFO:teuthology.misc.health.smithi088.stdout:{"epoch":10,"fsid":"869f5188-f92b-42dd-a46d-8c646e0388a9","created":"2024-03-29T10:40:40.285087+0000","modified":"2024-03-29T10:40:42.536230+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:40:42.536230+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a26c0c1c-8de0-438c-a3f6-3df10ea61574","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5de24915-8dec-49b6-8b16-f3170197c591","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ac054af3-d589-4565-9644-e6af75975c53","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3ec4f045-c714-4a22-bd1a-e07330b45ab3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"cddb9867-8e30-4951-9b13-3d1e78b9c45a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dc6d9911-785d-432f-8b53-56a1ccab302f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"77c88437-b587-496e-8b4d-1824ea8b5305","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ce963d96-5a8e-44c3-99de-c0dbebe234ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:43:03.472 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:03.488+0000 7f183759e640 10 librados: watch_flush enter 2024-03-29T10:43:03.472 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:03.488+0000 7f183759e640 10 librados: watch_flush exit 2024-03-29T10:43:03.472 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:03.488+0000 7f183759e640 20 client.4325.objecter shutdown clearing up homeless session... 2024-03-29T10:43:03.472 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:03.488+0000 7f183759e640 10 client.4325.objecter successfully canceled tick 2024-03-29T10:43:03.472 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:03.488+0000 7f183759e640 1 -- 172.21.15.88:0/1944034523 >> v1:172.21.15.88:6816/1752861349 conn(0x7f181806db00 legacy=0x7f181806ffc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:03.472 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:03.488+0000 7f183759e640 10 monclient: shutdown 2024-03-29T10:43:03.472 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:03.488+0000 7f183759e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:43:03.472 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:03.488+0000 7f183759e640 1 -- 172.21.15.88:0/1944034523 >> v1:172.21.15.88:6790/0 conn(0x7f18300ca8f0 legacy=0x7f1830167300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:03.472 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:03.488+0000 7f183759e640 1 -- 172.21.15.88:0/1944034523 shutdown_connections 2024-03-29T10:43:03.472 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:03.488+0000 7f183759e640 1 -- 172.21.15.88:0/1944034523 >> 172.21.15.88:0/1944034523 conn(0x7f1830005c00 msgr2=0x7f18300c4770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:43:03.472 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:03.488+0000 7f183759e640 1 -- 172.21.15.88:0/1944034523 shutdown_connections 2024-03-29T10:43:03.472 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:03.488+0000 7f183759e640 1 -- 172.21.15.88:0/1944034523 wait complete. 2024-03-29T10:43:03.472 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:03.489+0000 7f183759e640 1 librados: shutdown 2024-03-29T10:43:03.485 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:43:10.293 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:43:10.430 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:10.445+0000 7f423e086640 10 monclient: get_monmap_and_config 2024-03-29T10:43:10.430 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:10.445+0000 7f423e086640 10 monclient: build_initial_monmap 2024-03-29T10:43:10.430 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:10.445+0000 7f423e086640 10 monclient: monmap: 2024-03-29T10:43:10.430 INFO:teuthology.misc.health.smithi088.stderr:epoch 0 2024-03-29T10:43:10.430 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:43:10.430 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:43:10.446981+0000 2024-03-29T10:43:10.431 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:43:10.446981+0000 2024-03-29T10:43:10.431 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 0 (unknown) 2024-03-29T10:43:10.431 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:43:10.431 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-03-29T10:43:10.431 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-29T10:43:10.431 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-b 2024-03-29T10:43:10.431 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:43:10.431 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:10.447+0000 7f423e086640 1 Processor -- start 2024-03-29T10:43:10.431 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:10.447+0000 7f423e086640 1 -- start start 2024-03-29T10:43:10.431 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:10.447+0000 7f423e086640 10 monclient: init 2024-03-29T10:43:10.432 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:10.447+0000 7f423e086640 10 monclient: _reopen_session rank -1 2024-03-29T10:43:10.432 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:10.447+0000 7f423e086640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:43:10.432 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:10.447+0000 7f423e086640 10 monclient(hunting): picked mon.noname-c con 0x7f42380b0860 addr v1:172.21.15.196:6789/0 2024-03-29T10:43:10.432 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:10.447+0000 7f423e086640 10 monclient(hunting): picked mon.noname-b con 0x7f4238072450 addr v1:172.21.15.88:6790/0 2024-03-29T10:43:10.432 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:10.447+0000 7f423e086640 10 monclient(hunting): picked mon.noname-a con 0x7f423813ee40 addr v1:172.21.15.88:6789/0 2024-03-29T10:43:10.433 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:10.447+0000 7f423e086640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42380ad840 con 0x7f423813ee40 2024-03-29T10:43:10.433 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:10.447+0000 7f423e086640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f423813c830 con 0x7f42380b0860 2024-03-29T10:43:10.433 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:10.447+0000 7f423e086640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f423813c9c0 con 0x7f4238072450 2024-03-29T10:43:10.433 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:10.448+0000 7f423e086640 10 monclient(hunting): _renew_subs 2024-03-29T10:43:10.433 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:10.448+0000 7f423e086640 10 monclient(hunting): authenticate will time out at 912.683900s 2024-03-29T10:43:10.433 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:10.448+0000 7f42377fe640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f42380b0860 0x7f42380acfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.88:44646/0 (socket says 172.21.15.88:44646) 2024-03-29T10:43:10.433 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:10.448+0000 7f42377fe640 1 -- 172.21.15.88:0/3895078892 learned_addr learned my addr 172.21.15.88:0/3895078892 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-29T10:43:10.433 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:10.448+0000 7f42367fc640 1 -- 172.21.15.88:0/3895078892 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3166554386 0 0) 0x7f423813c9c0 con 0x7f4238072450 2024-03-29T10:43:10.433 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:10.448+0000 7f42367fc640 10 monclient(hunting): _negotiate 2024-03-29T10:43:10.433 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:10.448+0000 7f42367fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:43:10.433 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:10.448+0000 7f42367fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:43:10.433 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:10.448+0000 7f42367fc640 10 monclient(hunting): my global_id is 4328 2024-03-29T10:43:10.433 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:10.448+0000 7f42367fc640 1 -- 172.21.15.88:0/3895078892 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4220003520 con 0x7f4238072450 2024-03-29T10:43:10.433 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:10.448+0000 7f42367fc640 1 -- 172.21.15.88:0/3895078892 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2314766172 0 0) 0x7f423813c830 con 0x7f42380b0860 2024-03-29T10:43:10.433 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:10.448+0000 7f42367fc640 10 monclient(hunting): _negotiate 2024-03-29T10:43:10.434 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:10.448+0000 7f42367fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:43:10.434 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:10.448+0000 7f42367fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:43:10.434 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:10.448+0000 7f42367fc640 10 monclient(hunting): my global_id is 4306 2024-03-29T10:43:10.434 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:10.448+0000 7f42367fc640 1 -- 172.21.15.88:0/3895078892 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f423813c9c0 con 0x7f42380b0860 2024-03-29T10:43:10.434 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:10.448+0000 7f42367fc640 1 -- 172.21.15.88:0/3895078892 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3018735862 0 0) 0x7f4220003520 con 0x7f4238072450 2024-03-29T10:43:10.434 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:10.448+0000 7f42367fc640 1 -- 172.21.15.88:0/3895078892 >> v1:172.21.15.196:6789/0 conn(0x7f42380b0860 legacy=0x7f42380acfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:10.434 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:10.448+0000 7f42367fc640 1 -- 172.21.15.88:0/3895078892 >> v1:172.21.15.88:6789/0 conn(0x7f423813ee40 legacy=0x7f423813f260 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:43:10.438 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:10.451+0000 7f42367fc640 10 monclient: _finish_hunting 0 2024-03-29T10:43:10.438 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:10.451+0000 7f42367fc640 1 monclient: found mon.noname-b 2024-03-29T10:43:10.438 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:10.451+0000 7f42367fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:43:10.438 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:10.451+0000 7f42367fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.88:6790/0 2024-03-29T10:43:10.438 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:10.451+0000 7f42367fc640 1 -- 172.21.15.88:0/3895078892 --> v1:172.21.15.88:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f423813f970 con 0x7f4238072450 2024-03-29T10:43:10.438 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:10.451+0000 7f42367fc640 10 monclient: _finish_auth 0 2024-03-29T10:43:10.438 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:10.451+0000 7f42367fc640 10 monclient: _check_auth_tickets 2024-03-29T10:43:10.438 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:10.451+0000 7f42367fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:43:10.438 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:10.451+0000 7f42367fc640 1 -- 172.21.15.88:0/3895078892 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f422c001690 con 0x7f4238072450 2024-03-29T10:43:10.438 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:10.451+0000 7f42367fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:43:10.438 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:10.451+0000 7f42367fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:43:10.438 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:10.451+0000 7f42367fc640 10 monclient: dump: 2024-03-29T10:43:10.439 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:43:10.439 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:43:10.439 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:43:10.439 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:43:10.439 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:43:10.439 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:43:10.439 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:43:10.439 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:43:10.439 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:43:10.439 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:43:10.439 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:10.451+0000 7f423e086640 5 monclient: authenticate success, global_id 4328 2024-03-29T10:43:10.439 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:10.451+0000 7f423e086640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:43:10.439 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:10.451+0000 7f42367fc640 1 -- 172.21.15.88:0/3895078892 <== mon.2 v1:172.21.15.88:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f422c0018b0 con 0x7f4238072450 2024-03-29T10:43:10.439 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:10.451+0000 7f42367fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:43:10.440 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:10.451+0000 7f42367fc640 1 -- 172.21.15.88:0/3895078892 <== mon.2 v1:172.21.15.88:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f422c001c40 con 0x7f4238072450 2024-03-29T10:43:10.440 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:10.451+0000 7f42367fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:43:10.440 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:10.451+0000 7f42367fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:43:10.440 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:10.451+0000 7f42367fc640 10 monclient: dump: 2024-03-29T10:43:10.440 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:43:10.440 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:43:10.440 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:43:10.440 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:43:10.440 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:43:10.440 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:43:10.440 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:43:10.440 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:43:10.440 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:43:10.440 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:43:10.440 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:10.451+0000 7f423e086640 10 monclient: get_monmap_and_config success 2024-03-29T10:43:10.441 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:10.451+0000 7f423e086640 4 set_mon_vals no callback set 2024-03-29T10:43:10.441 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:10.453+0000 7f423e086640 10 monclient: shutdown 2024-03-29T10:43:10.441 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:10.453+0000 7f423e086640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:43:10.441 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:10.453+0000 7f423e086640 1 -- 172.21.15.88:0/3895078892 >> v1:172.21.15.88:6790/0 conn(0x7f4238072450 legacy=0x7f423813c120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:10.441 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:10.453+0000 7f423e086640 1 -- 172.21.15.88:0/3895078892 shutdown_connections 2024-03-29T10:43:10.441 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:10.453+0000 7f423e086640 1 -- 172.21.15.88:0/3895078892 >> 172.21.15.88:0/3895078892 conn(0x7f423806b660 msgr2=0x7f423806ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:43:10.441 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:10.453+0000 7f423e086640 1 -- 172.21.15.88:0/3895078892 shutdown_connections 2024-03-29T10:43:10.441 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:10.453+0000 7f423e086640 1 -- 172.21.15.88:0/3895078892 wait complete. 2024-03-29T10:43:10.441 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:10.454+0000 7f423e086640 10 monclient: build_initial_monmap 2024-03-29T10:43:10.441 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:10.454+0000 7f423e086640 10 monclient: monmap: 2024-03-29T10:43:10.441 INFO:teuthology.misc.health.smithi088.stderr:epoch 0 2024-03-29T10:43:10.441 INFO:teuthology.misc.health.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:43:10.441 INFO:teuthology.misc.health.smithi088.stderr:last_changed 0.000000 2024-03-29T10:43:10.441 INFO:teuthology.misc.health.smithi088.stderr:created 0.000000 2024-03-29T10:43:10.441 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 0 (unknown) 2024-03-29T10:43:10.442 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:43:10.442 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-03-29T10:43:10.442 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-29T10:43:10.442 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-c 2024-03-29T10:43:10.442 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:43:10.442 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:10.454+0000 7f423e086640 1 librados: starting msgr at 2024-03-29T10:43:10.442 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:10.454+0000 7f423e086640 1 librados: starting objecter 2024-03-29T10:43:10.442 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:10.454+0000 7f423e086640 1 Processor -- start 2024-03-29T10:43:10.447 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:10.455+0000 7f423e086640 1 -- start start 2024-03-29T10:43:10.447 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:10.455+0000 7f423e086640 1 librados: setting wanted keys 2024-03-29T10:43:10.447 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:10.455+0000 7f423e086640 1 librados: calling monclient init 2024-03-29T10:43:10.447 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:10.455+0000 7f423e086640 10 monclient: init 2024-03-29T10:43:10.447 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:10.457+0000 7f423e086640 10 monclient: _reopen_session rank -1 2024-03-29T10:43:10.447 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:10.457+0000 7f423e086640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:43:10.448 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:10.457+0000 7f423e086640 10 monclient(hunting): picked mon.noname-c con 0x7f4238072450 addr v1:172.21.15.88:6790/0 2024-03-29T10:43:10.448 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:10.457+0000 7f42377fe640 1 --1- >> v1:172.21.15.88:6790/0 conn(0x7f4238072450 0x7f42381d2e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6790/0 says I am v1:172.21.15.88:33626/0 (socket says 172.21.15.88:33626) 2024-03-29T10:43:10.448 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:10.457+0000 7f42377fe640 1 -- 172.21.15.88:0/418740679 learned_addr learned my addr 172.21.15.88:0/418740679 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-29T10:43:10.448 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:10.457+0000 7f421ffff640 10 client.?.objecter ms_handle_connect 0x7f4238072450 2024-03-29T10:43:10.448 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:10.457+0000 7f421ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:43:10.448 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:10.457+0000 7f423e086640 10 monclient(hunting): picked mon.noname-a con 0x7f42380b0860 addr v1:172.21.15.88:6789/0 2024-03-29T10:43:10.448 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:10.457+0000 7f423e086640 10 monclient(hunting): picked mon.noname-b con 0x7f423813ee40 addr v1:172.21.15.196:6789/0 2024-03-29T10:43:10.448 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:10.457+0000 7f423e086640 1 -- 172.21.15.88:0/418740679 --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42380b1d70 con 0x7f42380b0860 2024-03-29T10:43:10.448 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:10.457+0000 7f423e086640 1 -- 172.21.15.88:0/418740679 --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42380b32f0 con 0x7f423813ee40 2024-03-29T10:43:10.448 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:10.457+0000 7f423e086640 1 -- 172.21.15.88:0/418740679 --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4238134360 con 0x7f4238072450 2024-03-29T10:43:10.448 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:10.457+0000 7f423e086640 10 monclient(hunting): _renew_subs 2024-03-29T10:43:10.448 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:10.458+0000 7f423e086640 10 monclient(hunting): authenticate will time out at 912.693971s 2024-03-29T10:43:10.449 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:10.458+0000 7f421ffff640 1 -- 172.21.15.88:0/418740679 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4199626520 0 0) 0x7f4238134360 con 0x7f4238072450 2024-03-29T10:43:10.449 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:10.458+0000 7f421ffff640 10 monclient(hunting): _negotiate 2024-03-29T10:43:10.449 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:10.458+0000 7f421ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:43:10.449 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:10.458+0000 7f421ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:43:10.449 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:10.458+0000 7f421ffff640 10 monclient(hunting): my global_id is 4331 2024-03-29T10:43:10.449 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:10.458+0000 7f421ffff640 1 -- 172.21.15.88:0/418740679 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4224004eb0 con 0x7f4238072450 2024-03-29T10:43:10.449 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:10.458+0000 7f421ffff640 1 -- 172.21.15.88:0/418740679 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3315597267 0 0) 0x7f4224004eb0 con 0x7f4238072450 2024-03-29T10:43:10.449 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:10.458+0000 7f421ffff640 1 -- 172.21.15.88:0/418740679 --> v1:172.21.15.88:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4238134360 con 0x7f4238072450 2024-03-29T10:43:10.449 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:10.458+0000 7f421ffff640 1 -- 172.21.15.88:0/418740679 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f42280019f0 con 0x7f4238072450 2024-03-29T10:43:10.449 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:10.458+0000 7f421ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:43:10.449 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:10.458+0000 7f421ffff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:43:10.449 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:10.458+0000 7f421ffff640 10 monclient(hunting): dump: 2024-03-29T10:43:10.449 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:43:10.449 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:43:10.450 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:43:10.450 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:43:10.450 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:43:10.450 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:43:10.450 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:43:10.450 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:43:10.450 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:43:10.450 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:43:10.450 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:10.458+0000 7f421ffff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:43:10.451 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:10.458+0000 7f421ffff640 1 -- 172.21.15.88:0/418740679 <== mon.2 v1:172.21.15.88:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2674412645 0 0) 0x7f4238134360 con 0x7f4238072450 2024-03-29T10:43:10.451 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:10.458+0000 7f421ffff640 1 -- 172.21.15.88:0/418740679 >> v1:172.21.15.196:6789/0 conn(0x7f423813ee40 legacy=0x7f4238133c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:10.451 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:10.459+0000 7f421ffff640 1 -- 172.21.15.88:0/418740679 >> v1:172.21.15.88:6789/0 conn(0x7f42380b0860 legacy=0x7f4238133540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:43:10.451 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:10.459+0000 7f421ffff640 10 monclient: _finish_hunting 0 2024-03-29T10:43:10.451 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:10.459+0000 7f421ffff640 1 monclient: found mon.c 2024-03-29T10:43:10.451 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:10.459+0000 7f421ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:43:10.451 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:10.459+0000 7f421ffff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-03-29T10:43:10.451 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:10.460+0000 7f421ffff640 1 -- 172.21.15.88:0/418740679 --> v1:172.21.15.88:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42380b2910 con 0x7f4238072450 2024-03-29T10:43:10.451 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:10.460+0000 7f421ffff640 10 monclient: _finish_auth 0 2024-03-29T10:43:10.451 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:10.460+0000 7f421ffff640 10 monclient: _check_auth_tickets 2024-03-29T10:43:10.451 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:10.460+0000 7f421ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:43:10.451 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:10.460+0000 7f423e086640 5 monclient: authenticate success, global_id 4331 2024-03-29T10:43:10.451 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:10.460+0000 7f423e086640 10 monclient: _renew_subs 2024-03-29T10:43:10.451 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:10.460+0000 7f423e086640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-03-29T10:43:10.451 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:10.460+0000 7f423e086640 1 -- 172.21.15.88:0/418740679 --> v1:172.21.15.88:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42380b2bf0 con 0x7f4238072450 2024-03-29T10:43:10.452 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:10.460+0000 7f421ffff640 1 -- 172.21.15.88:0/418740679 <== mon.2 v1:172.21.15.88:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4228001b50 con 0x7f4238072450 2024-03-29T10:43:10.452 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:10.460+0000 7f423e086640 10 client.4331.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:43:10.452 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:10.460+0000 7f423e086640 10 monclient: _renew_subs 2024-03-29T10:43:10.452 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:10.460+0000 7f423e086640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-03-29T10:43:10.452 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:10.460+0000 7f423e086640 1 -- 172.21.15.88:0/418740679 --> v1:172.21.15.88:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f42380b1980 con 0x7f4238072450 2024-03-29T10:43:10.452 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:10.460+0000 7f423e086640 1 librados: init done 2024-03-29T10:43:10.452 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:10.460+0000 7f421ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:43:10.452 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:10.460+0000 7f421ffff640 1 -- 172.21.15.88:0/418740679 <== mon.2 v1:172.21.15.88:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f4228004c30 con 0x7f4238072450 2024-03-29T10:43:10.452 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:10.460+0000 7f421ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:43:10.452 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:10.460+0000 7f421ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:43:10.452 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:10.460+0000 7f421ffff640 10 monclient: dump: 2024-03-29T10:43:10.452 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:43:10.452 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:43:10.452 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:43:10.452 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:43:10.453 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:43:10.453 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:43:10.453 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:43:10.453 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:43:10.453 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:43:10.453 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:43:10.453 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:10.460+0000 7f4235ffb640 4 set_mon_vals no callback set 2024-03-29T10:43:10.453 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:10.461+0000 7f423e086640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:43:10.453 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:10.462+0000 7f423e086640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:43:10.453 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:10.462+0000 7f423e086640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-03-29T10:43:10.453 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:10.462+0000 7f423e086640 1 -- 172.21.15.88:0/418740679 --> v1:172.21.15.88:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4218005080 con 0x7f4238072450 2024-03-29T10:43:10.453 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:10.463+0000 7f421ffff640 1 -- 172.21.15.88:0/418740679 <== mon.2 v1:172.21.15.88:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4235629220 0 0) 0x7f422801ac00 con 0x7f4238072450 2024-03-29T10:43:10.453 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:10.463+0000 7f421ffff640 1 -- 172.21.15.88:0/418740679 <== mon.2 v1:172.21.15.88:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4257673582 0 0) 0x7f4228084be0 con 0x7f4238072450 2024-03-29T10:43:10.454 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:10.463+0000 7f421ffff640 10 client.4331.objecter ms_dispatch 0x7f4238000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:43:10.454 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:10.463+0000 7f421ffff640 3 client.4331.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:43:10.454 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:10.463+0000 7f421ffff640 3 client.4331.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:43:10.454 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:10.463+0000 7f421ffff640 20 client.4331.objecter dump_active .. 0 homeless 2024-03-29T10:43:10.454 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:10.464+0000 7f4236ffd640 10 monclient: get_auth_request con 0x7f422406db70 auth_method 0 2024-03-29T10:43:10.454 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:10.464+0000 7f421ffff640 10 client.4331.objecter ms_handle_connect 0x7f422406db70 2024-03-29T10:43:10.454 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:10.470+0000 7f421ffff640 1 -- 172.21.15.88:0/418740679 <== mon.2 v1:172.21.15.88:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f42280504c0 con 0x7f4238072450 2024-03-29T10:43:10.454 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:10.470+0000 7f421ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:43:10.455 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:10.470+0000 7f421ffff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:43:10.760 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:10.776+0000 7f423e086640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:43:10.760 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:10.776+0000 7f423e086640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:43:10.760 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:10.776+0000 7f423e086640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-03-29T10:43:10.760 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:10.776+0000 7f423e086640 1 -- 172.21.15.88:0/418740679 --> v1:172.21.15.88:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4218005330 con 0x7f4238072450 2024-03-29T10:43:10.761 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:10.777+0000 7f421ffff640 1 -- 172.21.15.88:0/418740679 <== mon.2 v1:172.21.15.88:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 344506374) 0x7f42280523f0 con 0x7f4238072450 2024-03-29T10:43:10.761 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:10.777+0000 7f421ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:43:10.761 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:10.777+0000 7f421ffff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:43:10.761 INFO:teuthology.misc.health.smithi088.stdout: 2024-03-29T10:43:10.761 INFO:teuthology.misc.health.smithi088.stdout:{"epoch":10,"fsid":"869f5188-f92b-42dd-a46d-8c646e0388a9","created":"2024-03-29T10:40:40.285087+0000","modified":"2024-03-29T10:40:42.536230+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:40:42.536230+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a26c0c1c-8de0-438c-a3f6-3df10ea61574","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5de24915-8dec-49b6-8b16-f3170197c591","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ac054af3-d589-4565-9644-e6af75975c53","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3ec4f045-c714-4a22-bd1a-e07330b45ab3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"cddb9867-8e30-4951-9b13-3d1e78b9c45a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dc6d9911-785d-432f-8b53-56a1ccab302f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"77c88437-b587-496e-8b4d-1824ea8b5305","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ce963d96-5a8e-44c3-99de-c0dbebe234ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:43:10.764 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:10.780+0000 7f423e086640 10 librados: watch_flush enter 2024-03-29T10:43:10.764 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:10.780+0000 7f423e086640 10 librados: watch_flush exit 2024-03-29T10:43:10.764 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:10.780+0000 7f423e086640 20 client.4331.objecter shutdown clearing up homeless session... 2024-03-29T10:43:10.764 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:10.780+0000 7f423e086640 10 client.4331.objecter successfully canceled tick 2024-03-29T10:43:10.764 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:10.780+0000 7f423e086640 1 -- 172.21.15.88:0/418740679 >> v1:172.21.15.88:6816/1752861349 conn(0x7f422406db70 legacy=0x7f4224070030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:10.764 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:10.780+0000 7f423e086640 10 monclient: shutdown 2024-03-29T10:43:10.765 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:10.780+0000 7f423e086640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:43:10.765 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:10.780+0000 7f423e086640 1 -- 172.21.15.88:0/418740679 >> v1:172.21.15.88:6790/0 conn(0x7f4238072450 legacy=0x7f42381d2e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:10.765 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:10.780+0000 7f423e086640 1 -- 172.21.15.88:0/418740679 shutdown_connections 2024-03-29T10:43:10.765 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:10.780+0000 7f423e086640 1 -- 172.21.15.88:0/418740679 >> 172.21.15.88:0/418740679 conn(0x7f423806b660 msgr2=0x7f42380a9d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:43:10.765 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:10.781+0000 7f423e086640 1 -- 172.21.15.88:0/418740679 shutdown_connections 2024-03-29T10:43:10.765 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:10.781+0000 7f423e086640 1 -- 172.21.15.88:0/418740679 wait complete. 2024-03-29T10:43:10.765 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:10.781+0000 7f423e086640 1 librados: shutdown 2024-03-29T10:43:10.777 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:43:17.583 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:43:17.731 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:17.747+0000 7f157d1c8640 10 monclient: get_monmap_and_config 2024-03-29T10:43:17.731 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:17.747+0000 7f157d1c8640 10 monclient: build_initial_monmap 2024-03-29T10:43:17.731 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:17.747+0000 7f157d1c8640 10 monclient: monmap: 2024-03-29T10:43:17.731 INFO:teuthology.misc.health.smithi088.stderr:epoch 0 2024-03-29T10:43:17.731 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:43:17.731 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:43:17.748801+0000 2024-03-29T10:43:17.731 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:43:17.748801+0000 2024-03-29T10:43:17.731 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 0 (unknown) 2024-03-29T10:43:17.731 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:43:17.731 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-03-29T10:43:17.731 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-29T10:43:17.732 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-b 2024-03-29T10:43:17.732 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:43:17.732 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:17.749+0000 7f157d1c8640 1 Processor -- start 2024-03-29T10:43:17.733 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:17.749+0000 7f157d1c8640 1 -- start start 2024-03-29T10:43:17.733 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:17.749+0000 7f157d1c8640 10 monclient: init 2024-03-29T10:43:17.734 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:17.750+0000 7f157d1c8640 10 monclient: _reopen_session rank -1 2024-03-29T10:43:17.734 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:17.750+0000 7f157d1c8640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:43:17.735 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:17.750+0000 7f157d1c8640 10 monclient(hunting): picked mon.noname-b con 0x7f15780b2800 addr v1:172.21.15.88:6790/0 2024-03-29T10:43:17.735 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:17.750+0000 7f157d1c8640 10 monclient(hunting): picked mon.noname-c con 0x7f15780a9c00 addr v1:172.21.15.196:6789/0 2024-03-29T10:43:17.736 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:17.753+0000 7f157d1c8640 10 monclient(hunting): picked mon.noname-a con 0x7f15780b38c0 addr v1:172.21.15.88:6789/0 2024-03-29T10:43:17.736 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:17.753+0000 7f157d1c8640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1578141eb0 con 0x7f15780b38c0 2024-03-29T10:43:17.737 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:17.753+0000 7f157d1c8640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15780b22d0 con 0x7f15780a9c00 2024-03-29T10:43:17.737 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:17.753+0000 7f157d1c8640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1578142d30 con 0x7f15780b2800 2024-03-29T10:43:17.737 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:17.753+0000 7f157d1c8640 10 monclient(hunting): _renew_subs 2024-03-29T10:43:17.737 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:17.753+0000 7f157d1c8640 10 monclient(hunting): authenticate will time out at 919.989005s 2024-03-29T10:43:17.737 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:17.754+0000 7f156f7fe640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f15780a9c00 0x7f157813bb60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.88:41938/0 (socket says 172.21.15.88:41938) 2024-03-29T10:43:17.737 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:17.754+0000 7f156f7fe640 1 -- 172.21.15.88:0/1810097247 learned_addr learned my addr 172.21.15.88:0/1810097247 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-29T10:43:17.739 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:17.755+0000 7f15777fe640 1 -- 172.21.15.88:0/1810097247 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2186806768 0 0) 0x7f15780b22d0 con 0x7f15780a9c00 2024-03-29T10:43:17.739 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:17.755+0000 7f15777fe640 10 monclient(hunting): _negotiate 2024-03-29T10:43:17.739 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:17.755+0000 7f15777fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:43:17.739 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:17.755+0000 7f15777fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:43:17.739 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:17.755+0000 7f15777fe640 10 monclient(hunting): my global_id is 4309 2024-03-29T10:43:17.739 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:17.755+0000 7f15777fe640 1 -- 172.21.15.88:0/1810097247 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1564003520 con 0x7f15780a9c00 2024-03-29T10:43:17.740 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:17.755+0000 7f15777fe640 1 -- 172.21.15.88:0/1810097247 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3818552234 0 0) 0x7f1578141eb0 con 0x7f15780b38c0 2024-03-29T10:43:17.740 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:17.755+0000 7f15777fe640 10 monclient(hunting): _negotiate 2024-03-29T10:43:17.740 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:17.755+0000 7f15777fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:43:17.740 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:17.755+0000 7f15777fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:43:17.740 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:17.755+0000 7f15777fe640 10 monclient(hunting): my global_id is 4311 2024-03-29T10:43:17.740 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:17.755+0000 7f15777fe640 1 -- 172.21.15.88:0/1810097247 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f15780b22d0 con 0x7f15780b38c0 2024-03-29T10:43:17.740 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:17.755+0000 7f15777fe640 1 -- 172.21.15.88:0/1810097247 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1283491069 0 0) 0x7f1578142d30 con 0x7f15780b2800 2024-03-29T10:43:17.740 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:17.755+0000 7f15777fe640 10 monclient(hunting): _negotiate 2024-03-29T10:43:17.740 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:17.755+0000 7f15777fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:43:17.740 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:17.755+0000 7f15777fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:43:17.740 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:17.755+0000 7f15777fe640 10 monclient(hunting): my global_id is 4334 2024-03-29T10:43:17.740 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:17.755+0000 7f15777fe640 1 -- 172.21.15.88:0/1810097247 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1578141eb0 con 0x7f15780b2800 2024-03-29T10:43:17.740 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:17.755+0000 7f15777fe640 1 -- 172.21.15.88:0/1810097247 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2049038753 0 0) 0x7f15780b22d0 con 0x7f15780b38c0 2024-03-29T10:43:17.740 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:17.755+0000 7f15777fe640 1 -- 172.21.15.88:0/1810097247 >> v1:172.21.15.88:6790/0 conn(0x7f15780b2800 legacy=0x7f1578142620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:17.740 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:17.755+0000 7f15777fe640 1 -- 172.21.15.88:0/1810097247 >> v1:172.21.15.196:6789/0 conn(0x7f15780a9c00 legacy=0x7f157813bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:17.741 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:17.755+0000 7f15777fe640 10 monclient: _finish_hunting 0 2024-03-29T10:43:17.741 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:17.755+0000 7f15777fe640 1 monclient: found mon.noname-a 2024-03-29T10:43:17.741 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:17.755+0000 7f15777fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:43:17.741 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:17.755+0000 7f15777fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.88:6789/0 2024-03-29T10:43:17.741 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:17.755+0000 7f15777fe640 1 -- 172.21.15.88:0/1810097247 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15780aaf90 con 0x7f15780b38c0 2024-03-29T10:43:17.741 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:17.755+0000 7f15777fe640 10 monclient: _finish_auth 0 2024-03-29T10:43:17.741 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:17.755+0000 7f15777fe640 10 monclient: _check_auth_tickets 2024-03-29T10:43:17.741 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:17.755+0000 7f15777fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:43:17.741 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:17.755+0000 7f15777fe640 1 -- 172.21.15.88:0/1810097247 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f15700018a0 con 0x7f15780b38c0 2024-03-29T10:43:17.741 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:17.755+0000 7f15777fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:43:17.741 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:17.755+0000 7f15777fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:43:17.741 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:17.755+0000 7f15777fe640 10 monclient: dump: 2024-03-29T10:43:17.741 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:43:17.741 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:43:17.742 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:43:17.742 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:43:17.742 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:43:17.742 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:43:17.742 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:43:17.742 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:43:17.742 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:43:17.742 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:43:17.742 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:17.755+0000 7f15777fe640 1 -- 172.21.15.88:0/1810097247 <== mon.0 v1:172.21.15.88:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1570001a80 con 0x7f15780b38c0 2024-03-29T10:43:17.742 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:17.755+0000 7f15777fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:43:17.742 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:17.755+0000 7f15777fe640 1 -- 172.21.15.88:0/1810097247 <== mon.0 v1:172.21.15.88:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f1570001e10 con 0x7f15780b38c0 2024-03-29T10:43:17.742 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:17.755+0000 7f15777fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:43:17.742 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:17.755+0000 7f15777fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:43:17.743 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:17.755+0000 7f15777fe640 10 monclient: dump: 2024-03-29T10:43:17.743 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:43:17.743 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:43:17.743 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:43:17.743 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:43:17.743 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:43:17.743 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:43:17.743 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:43:17.743 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:43:17.743 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:43:17.743 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:43:17.743 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:17.756+0000 7f157d1c8640 5 monclient: authenticate success, global_id 4311 2024-03-29T10:43:17.743 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:17.756+0000 7f157d1c8640 10 monclient: get_monmap_and_config success 2024-03-29T10:43:17.743 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:17.756+0000 7f157d1c8640 4 set_mon_vals no callback set 2024-03-29T10:43:17.744 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:17.756+0000 7f157d1c8640 10 monclient: shutdown 2024-03-29T10:43:17.744 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:17.756+0000 7f157d1c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:43:17.745 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:17.756+0000 7f157d1c8640 1 -- 172.21.15.88:0/1810097247 >> v1:172.21.15.88:6789/0 conn(0x7f15780b38c0 legacy=0x7f157814d7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:17.745 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:17.756+0000 7f157d1c8640 1 -- 172.21.15.88:0/1810097247 shutdown_connections 2024-03-29T10:43:17.745 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:17.756+0000 7f157d1c8640 1 -- 172.21.15.88:0/1810097247 >> 172.21.15.88:0/1810097247 conn(0x7f1578066a00 msgr2=0x7f1578066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:43:17.745 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:17.760+0000 7f157d1c8640 1 -- 172.21.15.88:0/1810097247 shutdown_connections 2024-03-29T10:43:17.745 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:17.760+0000 7f157d1c8640 1 -- 172.21.15.88:0/1810097247 wait complete. 2024-03-29T10:43:17.747 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:17.761+0000 7f157d1c8640 10 monclient: build_initial_monmap 2024-03-29T10:43:17.747 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:17.761+0000 7f157d1c8640 10 monclient: monmap: 2024-03-29T10:43:17.748 INFO:teuthology.misc.health.smithi088.stderr:epoch 0 2024-03-29T10:43:17.748 INFO:teuthology.misc.health.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:43:17.748 INFO:teuthology.misc.health.smithi088.stderr:last_changed 0.000000 2024-03-29T10:43:17.748 INFO:teuthology.misc.health.smithi088.stderr:created 0.000000 2024-03-29T10:43:17.748 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 0 (unknown) 2024-03-29T10:43:17.748 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:43:17.748 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-03-29T10:43:17.748 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-29T10:43:17.748 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-c 2024-03-29T10:43:17.748 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:43:17.748 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:17.762+0000 7f157d1c8640 1 librados: starting msgr at 2024-03-29T10:43:17.748 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:17.762+0000 7f157d1c8640 1 librados: starting objecter 2024-03-29T10:43:17.748 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:17.762+0000 7f157d1c8640 1 Processor -- start 2024-03-29T10:43:17.749 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:17.762+0000 7f157d1c8640 1 -- start start 2024-03-29T10:43:17.749 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:17.762+0000 7f157d1c8640 1 librados: setting wanted keys 2024-03-29T10:43:17.749 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:17.762+0000 7f157d1c8640 1 librados: calling monclient init 2024-03-29T10:43:17.749 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:17.762+0000 7f157d1c8640 10 monclient: init 2024-03-29T10:43:17.749 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:17.763+0000 7f157d1c8640 10 monclient: _reopen_session rank -1 2024-03-29T10:43:17.749 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:17.763+0000 7f157d1c8640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:43:17.753 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:17.766+0000 7f157d1c8640 10 monclient(hunting): picked mon.noname-a con 0x7f15780a9c00 addr v1:172.21.15.88:6789/0 2024-03-29T10:43:17.753 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:17.769+0000 7f1577fff640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f15780a9c00 0x7f15781e4080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:34293/0 (socket says 172.21.15.88:34293) 2024-03-29T10:43:17.753 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:17.769+0000 7f1577fff640 1 -- 172.21.15.88:0/2651155294 learned_addr learned my addr 172.21.15.88:0/2651155294 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-29T10:43:17.753 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:17.769+0000 7f15757fa640 10 client.?.objecter ms_handle_connect 0x7f15780a9c00 2024-03-29T10:43:17.753 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:17.769+0000 7f15757fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:43:17.754 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:17.769+0000 7f15757fa640 10 client.?.objecter ms_handle_connect 0x7f15780b2800 2024-03-29T10:43:17.754 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:17.769+0000 7f15757fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:43:17.754 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:17.769+0000 7f157d1c8640 10 monclient(hunting): picked mon.noname-c con 0x7f15780b2800 addr v1:172.21.15.88:6790/0 2024-03-29T10:43:17.754 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:17.769+0000 7f157d1c8640 10 monclient(hunting): picked mon.noname-b con 0x7f15780b38c0 addr v1:172.21.15.196:6789/0 2024-03-29T10:43:17.754 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:17.769+0000 7f157d1c8640 1 -- 172.21.15.88:0/2651155294 --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15780ab7b0 con 0x7f15780a9c00 2024-03-29T10:43:17.754 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:17.769+0000 7f157d1c8640 1 -- 172.21.15.88:0/2651155294 --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15781e3b60 con 0x7f15780b38c0 2024-03-29T10:43:17.754 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:17.769+0000 7f157d1c8640 1 -- 172.21.15.88:0/2651155294 --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1578070000 con 0x7f15780b2800 2024-03-29T10:43:17.754 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:17.769+0000 7f157d1c8640 10 monclient(hunting): _renew_subs 2024-03-29T10:43:17.755 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:17.769+0000 7f157d1c8640 10 monclient(hunting): authenticate will time out at 920.005605s 2024-03-29T10:43:17.755 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:17.769+0000 7f15757fa640 1 -- 172.21.15.88:0/2651155294 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2401770164 0 0) 0x7f1578070000 con 0x7f15780b2800 2024-03-29T10:43:17.755 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:17.769+0000 7f15757fa640 10 monclient(hunting): _negotiate 2024-03-29T10:43:17.755 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:17.769+0000 7f15757fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:43:17.755 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:17.769+0000 7f15757fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:43:17.755 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:17.769+0000 7f15757fa640 10 monclient(hunting): my global_id is 4337 2024-03-29T10:43:17.755 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:17.769+0000 7f15757fa640 1 -- 172.21.15.88:0/2651155294 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1558004eb0 con 0x7f15780b2800 2024-03-29T10:43:17.755 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:17.769+0000 7f15757fa640 1 -- 172.21.15.88:0/2651155294 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 491674125 0 0) 0x7f15780ab7b0 con 0x7f15780a9c00 2024-03-29T10:43:17.755 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:17.769+0000 7f15757fa640 10 monclient(hunting): _negotiate 2024-03-29T10:43:17.755 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:17.769+0000 7f15757fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:43:17.755 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:17.769+0000 7f15757fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:43:17.756 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:17.769+0000 7f15757fa640 10 monclient(hunting): my global_id is 4314 2024-03-29T10:43:17.756 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:17.769+0000 7f15757fa640 1 -- 172.21.15.88:0/2651155294 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1578070000 con 0x7f15780a9c00 2024-03-29T10:43:17.756 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:17.770+0000 7f15757fa640 10 client.?.objecter ms_handle_connect 0x7f15780b38c0 2024-03-29T10:43:17.756 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:17.770+0000 7f15757fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:43:17.756 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:17.770+0000 7f15757fa640 1 -- 172.21.15.88:0/2651155294 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1749068466 0 0) 0x7f1558004eb0 con 0x7f15780b2800 2024-03-29T10:43:17.756 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:17.770+0000 7f15757fa640 1 -- 172.21.15.88:0/2651155294 --> v1:172.21.15.88:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f15780ab7b0 con 0x7f15780b2800 2024-03-29T10:43:17.756 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:17.770+0000 7f15757fa640 1 -- 172.21.15.88:0/2651155294 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f1568001950 con 0x7f15780b2800 2024-03-29T10:43:17.756 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:17.770+0000 7f15757fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:43:17.757 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:17.770+0000 7f15757fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:43:17.757 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:17.770+0000 7f15757fa640 10 monclient(hunting): dump: 2024-03-29T10:43:17.757 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:43:17.757 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:43:17.757 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:43:17.758 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:43:17.758 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:43:17.758 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:43:17.758 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:43:17.758 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:43:17.758 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:43:17.758 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:43:17.759 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:17.770+0000 7f15757fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:43:17.759 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:17.770+0000 7f15757fa640 1 -- 172.21.15.88:0/2651155294 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2894908688 0 0) 0x7f1578070000 con 0x7f15780a9c00 2024-03-29T10:43:17.759 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:17.770+0000 7f15757fa640 1 -- 172.21.15.88:0/2651155294 --> v1:172.21.15.88:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1558004eb0 con 0x7f15780a9c00 2024-03-29T10:43:17.759 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:17.770+0000 7f15757fa640 1 -- 172.21.15.88:0/2651155294 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f1560001950 con 0x7f15780a9c00 2024-03-29T10:43:17.759 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:17.770+0000 7f15757fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:43:17.759 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:17.770+0000 7f15757fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:43:17.759 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:17.770+0000 7f15757fa640 10 monclient(hunting): dump: 2024-03-29T10:43:17.759 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:43:17.760 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:43:17.760 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:43:17.760 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:43:17.760 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:43:17.760 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:43:17.761 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:43:17.761 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:43:17.761 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:43:17.761 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:43:17.761 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:17.770+0000 7f15757fa640 1 -- 172.21.15.88:0/2651155294 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3837648158 0 0) 0x7f15781e3b60 con 0x7f15780b38c0 2024-03-29T10:43:17.761 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:17.770+0000 7f15757fa640 10 monclient(hunting): _negotiate 2024-03-29T10:43:17.762 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:17.770+0000 7f15757fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:43:17.762 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:17.770+0000 7f15757fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:43:17.762 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:17.770+0000 7f15757fa640 10 monclient(hunting): my global_id is 4312 2024-03-29T10:43:17.762 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:17.770+0000 7f15757fa640 1 -- 172.21.15.88:0/2651155294 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1578070000 con 0x7f15780b38c0 2024-03-29T10:43:17.762 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:17.770+0000 7f15757fa640 1 -- 172.21.15.88:0/2651155294 <== mon.2 v1:172.21.15.88:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3392059449 0 0) 0x7f15780ab7b0 con 0x7f15780b2800 2024-03-29T10:43:17.764 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:17.770+0000 7f15757fa640 1 -- 172.21.15.88:0/2651155294 >> v1:172.21.15.196:6789/0 conn(0x7f15780b38c0 legacy=0x7f15781ee490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:17.764 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:17.770+0000 7f15757fa640 1 -- 172.21.15.88:0/2651155294 >> v1:172.21.15.88:6789/0 conn(0x7f15780a9c00 legacy=0x7f15781e4080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:17.764 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:17.770+0000 7f15757fa640 10 monclient: _finish_hunting 0 2024-03-29T10:43:17.764 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:17.770+0000 7f15757fa640 1 monclient: found mon.c 2024-03-29T10:43:17.764 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:17.770+0000 7f15757fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:43:17.764 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:17.770+0000 7f15757fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-03-29T10:43:17.764 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:17.770+0000 7f15757fa640 1 -- 172.21.15.88:0/2651155294 --> v1:172.21.15.88:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15781e4790 con 0x7f15780b2800 2024-03-29T10:43:17.765 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:17.770+0000 7f15757fa640 10 monclient: _finish_auth 0 2024-03-29T10:43:17.765 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:17.770+0000 7f15757fa640 10 monclient: _check_auth_tickets 2024-03-29T10:43:17.765 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:17.770+0000 7f15757fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:43:17.765 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:17.770+0000 7f157d1c8640 5 monclient: authenticate success, global_id 4337 2024-03-29T10:43:17.765 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:17.770+0000 7f157d1c8640 10 monclient: _renew_subs 2024-03-29T10:43:17.765 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:17.770+0000 7f157d1c8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-03-29T10:43:17.765 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:17.770+0000 7f157d1c8640 1 -- 172.21.15.88:0/2651155294 --> v1:172.21.15.88:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15781e4920 con 0x7f15780b2800 2024-03-29T10:43:17.765 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:17.770+0000 7f157d1c8640 10 client.4337.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:43:17.765 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:17.770+0000 7f157d1c8640 10 monclient: _renew_subs 2024-03-29T10:43:17.765 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:17.770+0000 7f157d1c8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-03-29T10:43:17.765 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:17.770+0000 7f157d1c8640 1 -- 172.21.15.88:0/2651155294 --> v1:172.21.15.88:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f15780713d0 con 0x7f15780b2800 2024-03-29T10:43:17.765 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:17.770+0000 7f157d1c8640 1 librados: init done 2024-03-29T10:43:17.765 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:17.771+0000 7f15757fa640 1 -- 172.21.15.88:0/2651155294 <== mon.2 v1:172.21.15.88:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1568001d60 con 0x7f15780b2800 2024-03-29T10:43:17.766 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:17.771+0000 7f15757fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:43:17.766 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:17.771+0000 7f15757fa640 1 -- 172.21.15.88:0/2651155294 <== mon.2 v1:172.21.15.88:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f1568004980 con 0x7f15780b2800 2024-03-29T10:43:17.766 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:17.771+0000 7f15757fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:43:17.766 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:17.771+0000 7f15757fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:43:17.766 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:17.771+0000 7f15757fa640 10 monclient: dump: 2024-03-29T10:43:17.766 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:43:17.766 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:43:17.766 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:43:17.766 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:43:17.766 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:43:17.766 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:43:17.766 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:43:17.766 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:43:17.766 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:43:17.767 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:43:17.767 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:17.772+0000 7f1576ffd640 4 set_mon_vals no callback set 2024-03-29T10:43:17.767 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:17.773+0000 7f157d1c8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:43:17.767 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:17.773+0000 7f157d1c8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:43:17.767 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:17.773+0000 7f157d1c8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-03-29T10:43:17.767 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:17.773+0000 7f157d1c8640 1 -- 172.21.15.88:0/2651155294 --> v1:172.21.15.88:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f155c005080 con 0x7f15780b2800 2024-03-29T10:43:17.767 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:17.774+0000 7f15757fa640 1 -- 172.21.15.88:0/2651155294 <== mon.2 v1:172.21.15.88:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4235629220 0 0) 0x7f156801a950 con 0x7f15780b2800 2024-03-29T10:43:17.767 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:17.775+0000 7f1577fff640 10 monclient: get_auth_request con 0x7f155806dfa0 auth_method 0 2024-03-29T10:43:17.767 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:17.775+0000 7f15757fa640 1 -- 172.21.15.88:0/2651155294 <== mon.2 v1:172.21.15.88:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4257673582 0 0) 0x7f15680849d0 con 0x7f15780b2800 2024-03-29T10:43:17.767 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:17.775+0000 7f15757fa640 10 client.4337.objecter ms_dispatch 0x7f1578000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:43:17.767 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:17.775+0000 7f15757fa640 3 client.4337.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:43:17.767 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:17.775+0000 7f15757fa640 3 client.4337.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:43:17.767 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:17.775+0000 7f15757fa640 20 client.4337.objecter dump_active .. 0 homeless 2024-03-29T10:43:17.768 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:17.776+0000 7f15757fa640 10 client.4337.objecter ms_handle_connect 0x7f155806dfa0 2024-03-29T10:43:17.768 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:17.782+0000 7f15757fa640 1 -- 172.21.15.88:0/2651155294 <== mon.2 v1:172.21.15.88:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f15680500e0 con 0x7f15780b2800 2024-03-29T10:43:17.768 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:17.782+0000 7f15757fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:43:17.768 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:17.782+0000 7f15757fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:43:18.063 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:18.080+0000 7f157d1c8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:43:18.063 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:18.080+0000 7f157d1c8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:43:18.063 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:18.080+0000 7f157d1c8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-03-29T10:43:18.063 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:18.080+0000 7f157d1c8640 1 -- 172.21.15.88:0/2651155294 --> v1:172.21.15.88:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f155c005330 con 0x7f15780b2800 2024-03-29T10:43:18.064 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:18.081+0000 7f15757fa640 1 -- 172.21.15.88:0/2651155294 <== mon.2 v1:172.21.15.88:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 344506374) 0x7f15680521e0 con 0x7f15780b2800 2024-03-29T10:43:18.064 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:18.081+0000 7f15757fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:43:18.064 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:18.081+0000 7f15757fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:43:18.064 INFO:teuthology.misc.health.smithi088.stdout: 2024-03-29T10:43:18.064 INFO:teuthology.misc.health.smithi088.stdout:{"epoch":10,"fsid":"869f5188-f92b-42dd-a46d-8c646e0388a9","created":"2024-03-29T10:40:40.285087+0000","modified":"2024-03-29T10:40:42.536230+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:40:42.536230+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a26c0c1c-8de0-438c-a3f6-3df10ea61574","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5de24915-8dec-49b6-8b16-f3170197c591","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ac054af3-d589-4565-9644-e6af75975c53","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3ec4f045-c714-4a22-bd1a-e07330b45ab3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"cddb9867-8e30-4951-9b13-3d1e78b9c45a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dc6d9911-785d-432f-8b53-56a1ccab302f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"77c88437-b587-496e-8b4d-1824ea8b5305","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ce963d96-5a8e-44c3-99de-c0dbebe234ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:43:18.069 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:18.084+0000 7f157d1c8640 10 librados: watch_flush enter 2024-03-29T10:43:18.069 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:18.084+0000 7f157d1c8640 10 librados: watch_flush exit 2024-03-29T10:43:18.069 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:18.085+0000 7f157d1c8640 20 client.4337.objecter shutdown clearing up homeless session... 2024-03-29T10:43:18.069 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:18.085+0000 7f157d1c8640 10 client.4337.objecter successfully canceled tick 2024-03-29T10:43:18.069 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:18.085+0000 7f157d1c8640 1 -- 172.21.15.88:0/2651155294 >> v1:172.21.15.88:6816/1752861349 conn(0x7f155806dfa0 legacy=0x7f1558070460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:18.069 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:18.085+0000 7f157d1c8640 10 monclient: shutdown 2024-03-29T10:43:18.069 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:18.085+0000 7f157d1c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:43:18.069 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:18.085+0000 7f157d1c8640 1 -- 172.21.15.88:0/2651155294 >> v1:172.21.15.88:6790/0 conn(0x7f15780b2800 legacy=0x7f1578070ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:18.069 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:18.085+0000 7f157d1c8640 1 -- 172.21.15.88:0/2651155294 shutdown_connections 2024-03-29T10:43:18.069 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:18.085+0000 7f157d1c8640 1 -- 172.21.15.88:0/2651155294 >> 172.21.15.88:0/2651155294 conn(0x7f1578066a00 msgr2=0x7f157813bb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:43:18.069 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:18.086+0000 7f157d1c8640 1 -- 172.21.15.88:0/2651155294 shutdown_connections 2024-03-29T10:43:18.069 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:18.086+0000 7f157d1c8640 1 -- 172.21.15.88:0/2651155294 wait complete. 2024-03-29T10:43:18.069 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:18.086+0000 7f157d1c8640 1 librados: shutdown 2024-03-29T10:43:18.087 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:43:24.894 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:43:25.044 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:25.061+0000 7f063e5d6640 10 monclient: get_monmap_and_config 2024-03-29T10:43:25.044 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:25.061+0000 7f063e5d6640 10 monclient: build_initial_monmap 2024-03-29T10:43:25.044 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:25.061+0000 7f063e5d6640 10 monclient: monmap: 2024-03-29T10:43:25.044 INFO:teuthology.misc.health.smithi088.stderr:epoch 0 2024-03-29T10:43:25.044 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:43:25.044 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:43:25.062165+0000 2024-03-29T10:43:25.044 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:43:25.062165+0000 2024-03-29T10:43:25.044 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 0 (unknown) 2024-03-29T10:43:25.044 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:43:25.045 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-03-29T10:43:25.045 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-29T10:43:25.045 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-b 2024-03-29T10:43:25.045 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:43:25.045 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:25.062+0000 7f063e5d6640 1 Processor -- start 2024-03-29T10:43:25.047 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:25.062+0000 7f063e5d6640 1 -- start start 2024-03-29T10:43:25.047 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:25.062+0000 7f063e5d6640 10 monclient: init 2024-03-29T10:43:25.047 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:25.064+0000 7f063e5d6640 10 monclient: _reopen_session rank -1 2024-03-29T10:43:25.047 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:25.064+0000 7f063e5d6640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:43:25.047 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:25.064+0000 7f063e5d6640 10 monclient(hunting): picked mon.noname-a con 0x7f06381422b0 addr v1:172.21.15.88:6789/0 2024-03-29T10:43:25.048 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:25.064+0000 7f063e5d6640 10 monclient(hunting): picked mon.noname-c con 0x7f0638142690 addr v1:172.21.15.196:6789/0 2024-03-29T10:43:25.048 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:25.064+0000 7f063e5d6640 10 monclient(hunting): picked mon.noname-b con 0x7f06380b17d0 addr v1:172.21.15.88:6790/0 2024-03-29T10:43:25.048 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:25.064+0000 7f063e5d6640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0638142c80 con 0x7f06381422b0 2024-03-29T10:43:25.048 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:25.064+0000 7f063e5d6640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06380aae40 con 0x7f0638142690 2024-03-29T10:43:25.048 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:25.064+0000 7f063e5d6640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f063813fbd0 con 0x7f06380b17d0 2024-03-29T10:43:25.048 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:25.064+0000 7f063e5d6640 10 monclient(hunting): _renew_subs 2024-03-29T10:43:25.048 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:25.064+0000 7f063e5d6640 10 monclient(hunting): authenticate will time out at 927.300134s 2024-03-29T10:43:25.053 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:25.070+0000 7f063cb73640 1 --1- >> v1:172.21.15.88:6790/0 conn(0x7f06380b17d0 0x7f06380b2ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6790/0 says I am v1:172.21.15.88:52669/0 (socket says 172.21.15.88:52669) 2024-03-29T10:43:25.053 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:25.070+0000 7f063cb73640 1 -- 172.21.15.88:0/4010735858 learned_addr learned my addr 172.21.15.88:0/4010735858 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-29T10:43:25.053 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:25.070+0000 7f0636ffd640 1 -- 172.21.15.88:0/4010735858 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3131487037 0 0) 0x7f063813fbd0 con 0x7f06380b17d0 2024-03-29T10:43:25.053 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:25.070+0000 7f0636ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:43:25.053 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:25.070+0000 7f0636ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:43:25.054 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:25.070+0000 7f0636ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:43:25.054 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:25.070+0000 7f0636ffd640 10 monclient(hunting): my global_id is 4340 2024-03-29T10:43:25.054 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:25.070+0000 7f0636ffd640 1 -- 172.21.15.88:0/4010735858 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0620003520 con 0x7f06380b17d0 2024-03-29T10:43:25.054 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:25.071+0000 7f0636ffd640 1 -- 172.21.15.88:0/4010735858 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3397467391 0 0) 0x7f0620003520 con 0x7f06380b17d0 2024-03-29T10:43:25.054 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:25.071+0000 7f0636ffd640 1 -- 172.21.15.88:0/4010735858 >> v1:172.21.15.196:6789/0 conn(0x7f0638142690 legacy=0x7f06380b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:43:25.055 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:25.071+0000 7f0636ffd640 1 -- 172.21.15.88:0/4010735858 >> v1:172.21.15.88:6789/0 conn(0x7f06381422b0 legacy=0x7f063813f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:43:25.055 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:25.071+0000 7f0636ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:43:25.055 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:25.071+0000 7f0636ffd640 1 monclient: found mon.noname-b 2024-03-29T10:43:25.055 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:25.071+0000 7f0636ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:43:25.055 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:25.071+0000 7f0636ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.88:6790/0 2024-03-29T10:43:25.055 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:25.071+0000 7f0636ffd640 1 -- 172.21.15.88:0/4010735858 --> v1:172.21.15.88:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0638151940 con 0x7f06380b17d0 2024-03-29T10:43:25.055 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:25.071+0000 7f0636ffd640 10 monclient: _finish_auth 0 2024-03-29T10:43:25.055 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:25.071+0000 7f0636ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:43:25.055 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:25.071+0000 7f0636ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:43:25.055 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:25.071+0000 7f0636ffd640 1 -- 172.21.15.88:0/4010735858 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f0628001690 con 0x7f06380b17d0 2024-03-29T10:43:25.055 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:25.071+0000 7f0636ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:43:25.055 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:25.071+0000 7f0636ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:43:25.055 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:25.071+0000 7f0636ffd640 10 monclient: dump: 2024-03-29T10:43:25.055 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:43:25.056 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:43:25.056 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:43:25.056 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:43:25.056 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:43:25.056 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:43:25.056 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:43:25.056 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:43:25.056 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:43:25.056 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:43:25.056 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:25.071+0000 7f063e5d6640 5 monclient: authenticate success, global_id 4340 2024-03-29T10:43:25.056 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:25.071+0000 7f063e5d6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:43:25.056 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:25.071+0000 7f0636ffd640 1 -- 172.21.15.88:0/4010735858 <== mon.2 v1:172.21.15.88:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f06280017f0 con 0x7f06380b17d0 2024-03-29T10:43:25.056 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:25.071+0000 7f0636ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:43:25.056 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:25.071+0000 7f063e5d6640 10 monclient: get_monmap_and_config success 2024-03-29T10:43:25.056 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:25.071+0000 7f063e5d6640 4 set_mon_vals no callback set 2024-03-29T10:43:25.056 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:25.072+0000 7f0636ffd640 1 -- 172.21.15.88:0/4010735858 <== mon.2 v1:172.21.15.88:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f0628001c40 con 0x7f06380b17d0 2024-03-29T10:43:25.057 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:25.072+0000 7f0636ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:43:25.057 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:25.072+0000 7f0636ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:43:25.057 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:25.072+0000 7f0636ffd640 10 monclient: dump: 2024-03-29T10:43:25.057 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:43:25.057 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:43:25.057 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:43:25.057 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:43:25.057 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:43:25.057 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:43:25.057 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:43:25.057 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:43:25.057 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:43:25.057 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:43:25.059 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:25.072+0000 7f063e5d6640 10 monclient: shutdown 2024-03-29T10:43:25.059 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:25.072+0000 7f063e5d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:43:25.059 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:25.072+0000 7f063e5d6640 1 -- 172.21.15.88:0/4010735858 >> v1:172.21.15.88:6790/0 conn(0x7f06380b17d0 legacy=0x7f06380b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:25.059 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:25.072+0000 7f063e5d6640 1 -- 172.21.15.88:0/4010735858 shutdown_connections 2024-03-29T10:43:25.059 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:25.072+0000 7f063e5d6640 1 -- 172.21.15.88:0/4010735858 >> 172.21.15.88:0/4010735858 conn(0x7f0638066a00 msgr2=0x7f0638066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:43:25.059 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:25.073+0000 7f063e5d6640 1 -- 172.21.15.88:0/4010735858 shutdown_connections 2024-03-29T10:43:25.059 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:25.073+0000 7f063e5d6640 1 -- 172.21.15.88:0/4010735858 wait complete. 2024-03-29T10:43:25.059 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:25.074+0000 7f063e5d6640 10 monclient: build_initial_monmap 2024-03-29T10:43:25.059 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:25.074+0000 7f063e5d6640 10 monclient: monmap: 2024-03-29T10:43:25.059 INFO:teuthology.misc.health.smithi088.stderr:epoch 0 2024-03-29T10:43:25.060 INFO:teuthology.misc.health.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:43:25.060 INFO:teuthology.misc.health.smithi088.stderr:last_changed 0.000000 2024-03-29T10:43:25.060 INFO:teuthology.misc.health.smithi088.stderr:created 0.000000 2024-03-29T10:43:25.060 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 0 (unknown) 2024-03-29T10:43:25.060 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:43:25.060 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-03-29T10:43:25.060 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-29T10:43:25.060 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-c 2024-03-29T10:43:25.060 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:43:25.060 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:25.075+0000 7f063e5d6640 1 librados: starting msgr at 2024-03-29T10:43:25.060 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:25.075+0000 7f063e5d6640 1 librados: starting objecter 2024-03-29T10:43:25.060 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:25.075+0000 7f063e5d6640 1 Processor -- start 2024-03-29T10:43:25.060 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:25.075+0000 7f063e5d6640 1 -- start start 2024-03-29T10:43:25.060 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:25.075+0000 7f063e5d6640 1 librados: setting wanted keys 2024-03-29T10:43:25.060 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:25.075+0000 7f063e5d6640 1 librados: calling monclient init 2024-03-29T10:43:25.060 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:25.075+0000 7f063e5d6640 10 monclient: init 2024-03-29T10:43:25.062 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:25.076+0000 7f063e5d6640 10 monclient: _reopen_session rank -1 2024-03-29T10:43:25.062 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:25.076+0000 7f063e5d6640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:43:25.062 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:25.076+0000 7f063e5d6640 10 monclient(hunting): picked mon.noname-a con 0x7f06380b17d0 addr v1:172.21.15.88:6789/0 2024-03-29T10:43:25.062 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:25.076+0000 7f063e5d6640 10 monclient(hunting): picked mon.noname-b con 0x7f06381422b0 addr v1:172.21.15.196:6789/0 2024-03-29T10:43:25.062 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:25.076+0000 7f063e5d6640 10 monclient(hunting): picked mon.noname-c con 0x7f0638142690 addr v1:172.21.15.88:6790/0 2024-03-29T10:43:25.062 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:25.076+0000 7f063e5d6640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f063812f680 con 0x7f06380b17d0 2024-03-29T10:43:25.062 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:25.076+0000 7f063e5d6640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f063806e850 con 0x7f06381422b0 2024-03-29T10:43:25.062 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:25.076+0000 7f063e5d6640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f063806eb30 con 0x7f0638142690 2024-03-29T10:43:25.062 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:25.076+0000 7f063e5d6640 10 monclient(hunting): _renew_subs 2024-03-29T10:43:25.062 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:25.076+0000 7f063e5d6640 10 monclient(hunting): authenticate will time out at 927.312763s 2024-03-29T10:43:25.063 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:25.080+0000 7f0637fff640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f06380b17d0 0x7f0638150b10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:35269/0 (socket says 172.21.15.88:35269) 2024-03-29T10:43:25.064 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:25.081+0000 7f0637fff640 1 -- 172.21.15.88:0/2578788020 learned_addr learned my addr 172.21.15.88:0/2578788020 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-29T10:43:25.064 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:25.081+0000 7f0634ff9640 10 client.?.objecter ms_handle_connect 0x7f06380b17d0 2024-03-29T10:43:25.064 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:25.081+0000 7f0634ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:43:25.065 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:25.082+0000 7f0634ff9640 1 -- 172.21.15.88:0/2578788020 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1479421574 0 0) 0x7f063812f680 con 0x7f06380b17d0 2024-03-29T10:43:25.065 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:25.082+0000 7f0634ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:43:25.065 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:25.082+0000 7f0634ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:43:25.066 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:25.082+0000 7f0634ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:43:25.066 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:25.082+0000 7f0634ff9640 10 monclient(hunting): my global_id is 4317 2024-03-29T10:43:25.066 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:25.082+0000 7f0634ff9640 1 -- 172.21.15.88:0/2578788020 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0624004eb0 con 0x7f06380b17d0 2024-03-29T10:43:25.066 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:25.082+0000 7f0634ff9640 10 client.?.objecter ms_handle_connect 0x7f06381422b0 2024-03-29T10:43:25.066 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:25.082+0000 7f0634ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:43:25.066 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:25.082+0000 7f0634ff9640 1 -- 172.21.15.88:0/2578788020 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 579617448 0 0) 0x7f0624004eb0 con 0x7f06380b17d0 2024-03-29T10:43:25.066 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:25.082+0000 7f0634ff9640 1 -- 172.21.15.88:0/2578788020 --> v1:172.21.15.88:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f063812f680 con 0x7f06380b17d0 2024-03-29T10:43:25.068 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:25.085+0000 7f0634ff9640 10 client.?.objecter ms_handle_connect 0x7f0638142690 2024-03-29T10:43:25.068 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:25.085+0000 7f0634ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:43:25.070 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:25.086+0000 7f0634ff9640 1 -- 172.21.15.88:0/2578788020 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f062c001960 con 0x7f06380b17d0 2024-03-29T10:43:25.071 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:25.086+0000 7f0634ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:43:25.071 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:25.086+0000 7f0634ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:43:25.071 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:25.086+0000 7f0634ff9640 10 monclient(hunting): dump: 2024-03-29T10:43:25.071 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:43:25.071 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:43:25.071 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:43:25.071 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:43:25.071 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:43:25.071 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:43:25.071 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:43:25.071 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:43:25.071 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:43:25.071 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:43:25.071 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:25.086+0000 7f0634ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:43:25.072 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:25.086+0000 7f0634ff9640 1 -- 172.21.15.88:0/2578788020 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2432783733 0 0) 0x7f063806e850 con 0x7f06381422b0 2024-03-29T10:43:25.072 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:25.086+0000 7f0634ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:43:25.072 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:25.086+0000 7f0634ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:43:25.072 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:25.086+0000 7f0634ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:43:25.072 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:25.086+0000 7f0634ff9640 10 monclient(hunting): my global_id is 4315 2024-03-29T10:43:25.072 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:25.086+0000 7f0634ff9640 1 -- 172.21.15.88:0/2578788020 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0624004eb0 con 0x7f06381422b0 2024-03-29T10:43:25.072 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:25.086+0000 7f0634ff9640 1 -- 172.21.15.88:0/2578788020 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2522047733 0 0) 0x7f063806eb30 con 0x7f0638142690 2024-03-29T10:43:25.072 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:25.086+0000 7f0634ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:43:25.072 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:25.086+0000 7f0634ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:43:25.072 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:25.086+0000 7f0634ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:43:25.072 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:25.087+0000 7f0634ff9640 10 monclient(hunting): my global_id is 4343 2024-03-29T10:43:25.072 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:25.087+0000 7f0634ff9640 1 -- 172.21.15.88:0/2578788020 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f063806e850 con 0x7f0638142690 2024-03-29T10:43:25.072 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:25.087+0000 7f0634ff9640 1 -- 172.21.15.88:0/2578788020 <== mon.0 v1:172.21.15.88:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3051435892 0 0) 0x7f063812f680 con 0x7f06380b17d0 2024-03-29T10:43:25.072 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:25.087+0000 7f0634ff9640 1 -- 172.21.15.88:0/2578788020 >> v1:172.21.15.88:6790/0 conn(0x7f0638142690 legacy=0x7f063812ef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:25.072 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:25.087+0000 7f0634ff9640 1 -- 172.21.15.88:0/2578788020 >> v1:172.21.15.196:6789/0 conn(0x7f06381422b0 legacy=0x7f063812e860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:25.072 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:25.087+0000 7f0634ff9640 10 monclient: _finish_hunting 0 2024-03-29T10:43:25.072 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:25.087+0000 7f0634ff9640 1 monclient: found mon.a 2024-03-29T10:43:25.074 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:25.090+0000 7f0634ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:43:25.074 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:25.090+0000 7f0634ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-03-29T10:43:25.074 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:25.090+0000 7f0634ff9640 1 -- 172.21.15.88:0/2578788020 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f063806ee10 con 0x7f06380b17d0 2024-03-29T10:43:25.075 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:25.090+0000 7f0634ff9640 10 monclient: _finish_auth 0 2024-03-29T10:43:25.075 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:25.090+0000 7f0634ff9640 10 monclient: _check_auth_tickets 2024-03-29T10:43:25.075 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:25.090+0000 7f0634ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:43:25.075 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:25.090+0000 7f063e5d6640 5 monclient: authenticate success, global_id 4317 2024-03-29T10:43:25.075 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:25.090+0000 7f063e5d6640 10 monclient: _renew_subs 2024-03-29T10:43:25.075 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:25.090+0000 7f063e5d6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-03-29T10:43:25.075 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:25.090+0000 7f063e5d6640 1 -- 172.21.15.88:0/2578788020 --> v1:172.21.15.88:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0638150600 con 0x7f06380b17d0 2024-03-29T10:43:25.075 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:25.090+0000 7f063e5d6640 10 client.4317.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:43:25.075 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:25.090+0000 7f063e5d6640 10 monclient: _renew_subs 2024-03-29T10:43:25.075 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:25.090+0000 7f063e5d6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-03-29T10:43:25.075 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:25.090+0000 7f063e5d6640 1 -- 172.21.15.88:0/2578788020 --> v1:172.21.15.88:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f063806e400 con 0x7f06380b17d0 2024-03-29T10:43:25.075 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:25.090+0000 7f063e5d6640 1 librados: init done 2024-03-29T10:43:25.076 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:25.092+0000 7f063e5d6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:43:25.076 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:25.092+0000 7f063e5d6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:43:25.076 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:25.092+0000 7f063e5d6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-03-29T10:43:25.077 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:25.092+0000 7f063e5d6640 1 -- 172.21.15.88:0/2578788020 --> v1:172.21.15.88:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f063806e010 con 0x7f06380b17d0 2024-03-29T10:43:25.077 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:25.093+0000 7f0634ff9640 1 -- 172.21.15.88:0/2578788020 <== mon.0 v1:172.21.15.88:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f062c001b80 con 0x7f06380b17d0 2024-03-29T10:43:25.077 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:25.093+0000 7f0634ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:43:25.077 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:25.093+0000 7f0634ff9640 1 -- 172.21.15.88:0/2578788020 <== mon.0 v1:172.21.15.88:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f062c001d80 con 0x7f06380b17d0 2024-03-29T10:43:25.077 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:25.093+0000 7f0634ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:43:25.077 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:25.093+0000 7f0634ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:43:25.077 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:25.093+0000 7f0634ff9640 10 monclient: dump: 2024-03-29T10:43:25.077 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:43:25.077 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:43:25.077 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:43:25.077 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:43:25.077 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:43:25.077 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:43:25.077 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:43:25.077 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:43:25.078 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:43:25.078 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:43:25.078 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:25.093+0000 7f06367fc640 4 set_mon_vals no callback set 2024-03-29T10:43:25.082 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:25.098+0000 7f0634ff9640 1 -- 172.21.15.88:0/2578788020 <== mon.0 v1:172.21.15.88:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4235629220 0 0) 0x7f062c003a40 con 0x7f06380b17d0 2024-03-29T10:43:25.082 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:25.099+0000 7f0634ff9640 1 -- 172.21.15.88:0/2578788020 <== mon.0 v1:172.21.15.88:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4257673582 0 0) 0x7f062c084a40 con 0x7f06380b17d0 2024-03-29T10:43:25.082 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:25.099+0000 7f0634ff9640 10 client.4317.objecter ms_dispatch 0x7f0638000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:43:25.083 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:25.099+0000 7f0634ff9640 3 client.4317.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:43:25.083 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:25.099+0000 7f0634ff9640 3 client.4317.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:43:25.083 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:25.099+0000 7f0634ff9640 20 client.4317.objecter dump_active .. 0 homeless 2024-03-29T10:43:25.083 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:25.100+0000 7f06377fe640 10 monclient: get_auth_request con 0x7f062406de00 auth_method 0 2024-03-29T10:43:25.084 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:25.101+0000 7f0634ff9640 10 client.4317.objecter ms_handle_connect 0x7f062406de00 2024-03-29T10:43:25.085 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:25.102+0000 7f0634ff9640 1 -- 172.21.15.88:0/2578788020 <== mon.0 v1:172.21.15.88:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f062c050150 con 0x7f06380b17d0 2024-03-29T10:43:25.085 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:25.102+0000 7f0634ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:43:25.085 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:25.102+0000 7f0634ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:43:25.405 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:25.422+0000 7f063e5d6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:43:25.405 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:25.422+0000 7f063e5d6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:43:25.405 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:25.422+0000 7f063e5d6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-03-29T10:43:25.406 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:25.422+0000 7f063e5d6640 1 -- 172.21.15.88:0/2578788020 --> v1:172.21.15.88:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0638075f10 con 0x7f06380b17d0 2024-03-29T10:43:25.409 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:25.426+0000 7f0634ff9640 1 -- 172.21.15.88:0/2578788020 <== mon.0 v1:172.21.15.88:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 344506374) 0x7f062c052250 con 0x7f06380b17d0 2024-03-29T10:43:25.409 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:25.426+0000 7f0634ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:43:25.409 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:25.426+0000 7f0634ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:43:25.411 INFO:teuthology.misc.health.smithi088.stdout: 2024-03-29T10:43:25.411 INFO:teuthology.misc.health.smithi088.stdout:{"epoch":10,"fsid":"869f5188-f92b-42dd-a46d-8c646e0388a9","created":"2024-03-29T10:40:40.285087+0000","modified":"2024-03-29T10:40:42.536230+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:40:42.536230+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a26c0c1c-8de0-438c-a3f6-3df10ea61574","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5de24915-8dec-49b6-8b16-f3170197c591","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ac054af3-d589-4565-9644-e6af75975c53","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3ec4f045-c714-4a22-bd1a-e07330b45ab3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"cddb9867-8e30-4951-9b13-3d1e78b9c45a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dc6d9911-785d-432f-8b53-56a1ccab302f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"77c88437-b587-496e-8b4d-1824ea8b5305","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ce963d96-5a8e-44c3-99de-c0dbebe234ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:43:25.413 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:25.429+0000 7f063e5d6640 10 librados: watch_flush enter 2024-03-29T10:43:25.413 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:25.429+0000 7f063e5d6640 10 librados: watch_flush exit 2024-03-29T10:43:25.413 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:25.429+0000 7f063e5d6640 20 client.4317.objecter shutdown clearing up homeless session... 2024-03-29T10:43:25.413 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:25.429+0000 7f063e5d6640 10 client.4317.objecter successfully canceled tick 2024-03-29T10:43:25.413 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:25.430+0000 7f063e5d6640 1 -- 172.21.15.88:0/2578788020 >> v1:172.21.15.88:6816/1752861349 conn(0x7f062406de00 legacy=0x7f06240702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:25.413 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:25.430+0000 7f063e5d6640 10 monclient: shutdown 2024-03-29T10:43:25.413 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:25.430+0000 7f063e5d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:43:25.413 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:25.430+0000 7f063e5d6640 1 -- 172.21.15.88:0/2578788020 >> v1:172.21.15.88:6789/0 conn(0x7f06380b17d0 legacy=0x7f0638150b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:25.416 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:25.433+0000 7f063e5d6640 1 -- 172.21.15.88:0/2578788020 shutdown_connections 2024-03-29T10:43:25.416 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:25.433+0000 7f063e5d6640 1 -- 172.21.15.88:0/2578788020 >> 172.21.15.88:0/2578788020 conn(0x7f0638066a00 msgr2=0x7f0638131c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:43:25.416 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:25.433+0000 7f063e5d6640 1 -- 172.21.15.88:0/2578788020 shutdown_connections 2024-03-29T10:43:25.416 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:25.433+0000 7f063e5d6640 1 -- 172.21.15.88:0/2578788020 wait complete. 2024-03-29T10:43:25.416 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:25.433+0000 7f063e5d6640 1 librados: shutdown 2024-03-29T10:43:25.432 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:43:32.239 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:43:32.387 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:32.403+0000 7faf35f54640 10 monclient: get_monmap_and_config 2024-03-29T10:43:32.387 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:32.403+0000 7faf35f54640 10 monclient: build_initial_monmap 2024-03-29T10:43:32.387 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:32.403+0000 7faf35f54640 10 monclient: monmap: 2024-03-29T10:43:32.387 INFO:teuthology.misc.health.smithi088.stderr:epoch 0 2024-03-29T10:43:32.387 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:43:32.387 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:43:32.404795+0000 2024-03-29T10:43:32.387 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:43:32.404795+0000 2024-03-29T10:43:32.387 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 0 (unknown) 2024-03-29T10:43:32.387 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:43:32.388 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-03-29T10:43:32.388 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-29T10:43:32.388 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-b 2024-03-29T10:43:32.388 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:43:32.390 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:32.407+0000 7faf35f54640 1 Processor -- start 2024-03-29T10:43:32.393 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:32.408+0000 7faf35f54640 1 -- start start 2024-03-29T10:43:32.393 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:32.408+0000 7faf35f54640 10 monclient: init 2024-03-29T10:43:32.393 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:32.409+0000 7faf35f54640 10 monclient: _reopen_session rank -1 2024-03-29T10:43:32.393 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:32.409+0000 7faf35f54640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:43:32.393 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:32.409+0000 7faf35f54640 10 monclient(hunting): picked mon.noname-b con 0x7faf301457f0 addr v1:172.21.15.88:6790/0 2024-03-29T10:43:32.394 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:32.409+0000 7faf35f54640 10 monclient(hunting): picked mon.noname-a con 0x7faf3013f650 addr v1:172.21.15.88:6789/0 2024-03-29T10:43:32.394 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:32.409+0000 7faf35f54640 10 monclient(hunting): picked mon.noname-c con 0x7faf300a9d70 addr v1:172.21.15.196:6789/0 2024-03-29T10:43:32.394 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:32.409+0000 7faf35f54640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf300aaef0 con 0x7faf3013f650 2024-03-29T10:43:32.394 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:32.409+0000 7faf35f54640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf30145bd0 con 0x7faf300a9d70 2024-03-29T10:43:32.394 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:32.409+0000 7faf35f54640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf30145eb0 con 0x7faf301457f0 2024-03-29T10:43:32.394 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:32.409+0000 7faf35f54640 10 monclient(hunting): _renew_subs 2024-03-29T10:43:32.394 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:32.409+0000 7faf35f54640 10 monclient(hunting): authenticate will time out at 934.645116s 2024-03-29T10:43:32.396 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:32.412+0000 7faf2f7fe640 1 --1- >> v1:172.21.15.88:6790/0 conn(0x7faf301457f0 0x7faf3013ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6790/0 says I am v1:172.21.15.88:50875/0 (socket says 172.21.15.88:50875) 2024-03-29T10:43:32.396 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:32.413+0000 7faf2f7fe640 1 -- 172.21.15.88:0/3003741761 learned_addr learned my addr 172.21.15.88:0/3003741761 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-29T10:43:32.397 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:32.414+0000 7faf2e7fc640 1 -- 172.21.15.88:0/3003741761 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 275281586 0 0) 0x7faf30145eb0 con 0x7faf301457f0 2024-03-29T10:43:32.397 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:32.414+0000 7faf2e7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:43:32.397 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:32.414+0000 7faf2e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:43:32.397 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:32.414+0000 7faf2e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:43:32.397 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:32.414+0000 7faf2e7fc640 10 monclient(hunting): my global_id is 4346 2024-03-29T10:43:32.398 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:32.414+0000 7faf2e7fc640 1 -- 172.21.15.88:0/3003741761 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faf18003520 con 0x7faf301457f0 2024-03-29T10:43:32.399 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:32.416+0000 7faf2e7fc640 1 -- 172.21.15.88:0/3003741761 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1031089774 0 0) 0x7faf18003520 con 0x7faf301457f0 2024-03-29T10:43:32.399 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:32.416+0000 7faf2e7fc640 1 -- 172.21.15.88:0/3003741761 >> v1:172.21.15.196:6789/0 conn(0x7faf300a9d70 legacy=0x7faf3014d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:43:32.400 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:32.416+0000 7faf2e7fc640 1 -- 172.21.15.88:0/3003741761 >> v1:172.21.15.88:6789/0 conn(0x7faf3013f650 legacy=0x7faf300b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:32.400 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:32.416+0000 7faf2e7fc640 10 monclient: _finish_hunting 0 2024-03-29T10:43:32.400 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:32.416+0000 7faf2e7fc640 1 monclient: found mon.noname-b 2024-03-29T10:43:32.400 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:32.416+0000 7faf2e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:43:32.400 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:32.416+0000 7faf2e7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.88:6790/0 2024-03-29T10:43:32.400 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:32.416+0000 7faf2e7fc640 1 -- 172.21.15.88:0/3003741761 --> v1:172.21.15.88:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf30146340 con 0x7faf301457f0 2024-03-29T10:43:32.400 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:32.416+0000 7faf2e7fc640 10 monclient: _finish_auth 0 2024-03-29T10:43:32.401 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:32.416+0000 7faf2e7fc640 10 monclient: _check_auth_tickets 2024-03-29T10:43:32.401 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:32.416+0000 7faf2e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:43:32.401 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:32.416+0000 7faf2e7fc640 1 -- 172.21.15.88:0/3003741761 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7faf24001690 con 0x7faf301457f0 2024-03-29T10:43:32.401 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:32.416+0000 7faf2e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:43:32.401 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:32.416+0000 7faf2e7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:43:32.401 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:32.416+0000 7faf2e7fc640 10 monclient: dump: 2024-03-29T10:43:32.401 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:43:32.401 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:43:32.401 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:43:32.401 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:43:32.401 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:43:32.401 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:43:32.401 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:43:32.401 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:43:32.402 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:43:32.402 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:43:32.402 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:32.416+0000 7faf2e7fc640 1 -- 172.21.15.88:0/3003741761 <== mon.2 v1:172.21.15.88:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7faf24001860 con 0x7faf301457f0 2024-03-29T10:43:32.402 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:32.416+0000 7faf2e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:43:32.402 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:32.416+0000 7faf2e7fc640 1 -- 172.21.15.88:0/3003741761 <== mon.2 v1:172.21.15.88:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7faf24001c70 con 0x7faf301457f0 2024-03-29T10:43:32.402 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:32.416+0000 7faf2e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:43:32.402 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:32.416+0000 7faf2e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:43:32.402 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:32.416+0000 7faf2e7fc640 10 monclient: dump: 2024-03-29T10:43:32.402 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:43:32.402 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:43:32.402 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:43:32.402 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:43:32.402 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:43:32.402 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:43:32.402 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:43:32.403 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:43:32.403 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:43:32.403 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:43:32.403 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:32.417+0000 7faf35f54640 5 monclient: authenticate success, global_id 4346 2024-03-29T10:43:32.403 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:32.417+0000 7faf35f54640 10 monclient: get_monmap_and_config success 2024-03-29T10:43:32.403 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:32.417+0000 7faf35f54640 4 set_mon_vals no callback set 2024-03-29T10:43:32.405 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:32.418+0000 7faf35f54640 10 monclient: shutdown 2024-03-29T10:43:32.405 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:32.418+0000 7faf35f54640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:43:32.405 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:32.418+0000 7faf35f54640 1 -- 172.21.15.88:0/3003741761 >> v1:172.21.15.88:6790/0 conn(0x7faf301457f0 legacy=0x7faf3013ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:32.405 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:32.418+0000 7faf35f54640 1 -- 172.21.15.88:0/3003741761 shutdown_connections 2024-03-29T10:43:32.405 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:32.418+0000 7faf35f54640 1 -- 172.21.15.88:0/3003741761 >> 172.21.15.88:0/3003741761 conn(0x7faf30066a00 msgr2=0x7faf30066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:43:32.405 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:32.418+0000 7faf35f54640 1 -- 172.21.15.88:0/3003741761 shutdown_connections 2024-03-29T10:43:32.405 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:32.418+0000 7faf35f54640 1 -- 172.21.15.88:0/3003741761 wait complete. 2024-03-29T10:43:32.405 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:32.419+0000 7faf35f54640 10 monclient: build_initial_monmap 2024-03-29T10:43:32.405 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:32.419+0000 7faf35f54640 10 monclient: monmap: 2024-03-29T10:43:32.405 INFO:teuthology.misc.health.smithi088.stderr:epoch 0 2024-03-29T10:43:32.405 INFO:teuthology.misc.health.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:43:32.405 INFO:teuthology.misc.health.smithi088.stderr:last_changed 0.000000 2024-03-29T10:43:32.406 INFO:teuthology.misc.health.smithi088.stderr:created 0.000000 2024-03-29T10:43:32.406 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 0 (unknown) 2024-03-29T10:43:32.406 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:43:32.406 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-03-29T10:43:32.406 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-29T10:43:32.406 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-c 2024-03-29T10:43:32.406 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:43:32.406 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:32.420+0000 7faf35f54640 1 librados: starting msgr at 2024-03-29T10:43:32.406 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:32.421+0000 7faf35f54640 1 librados: starting objecter 2024-03-29T10:43:32.406 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:32.421+0000 7faf35f54640 1 Processor -- start 2024-03-29T10:43:32.406 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:32.421+0000 7faf35f54640 1 -- start start 2024-03-29T10:43:32.406 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:32.421+0000 7faf35f54640 1 librados: setting wanted keys 2024-03-29T10:43:32.406 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:32.421+0000 7faf35f54640 1 librados: calling monclient init 2024-03-29T10:43:32.406 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:32.421+0000 7faf35f54640 10 monclient: init 2024-03-29T10:43:32.407 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:32.422+0000 7faf35f54640 10 monclient: _reopen_session rank -1 2024-03-29T10:43:32.407 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:32.422+0000 7faf35f54640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:43:32.408 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:32.422+0000 7faf35f54640 10 monclient(hunting): picked mon.noname-c con 0x7faf300a9d70 addr v1:172.21.15.88:6790/0 2024-03-29T10:43:32.408 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:32.422+0000 7faf35f54640 10 monclient(hunting): picked mon.noname-b con 0x7faf3013f650 addr v1:172.21.15.196:6789/0 2024-03-29T10:43:32.412 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:32.425+0000 7faf2f7fe640 1 --1- >> v1:172.21.15.88:6790/0 conn(0x7faf300a9d70 0x7faf30136f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6790/0 says I am v1:172.21.15.88:50955/0 (socket says 172.21.15.88:50955) 2024-03-29T10:43:32.412 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:32.426+0000 7faf2ffff640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7faf301457f0 0x7faf301d6ad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:57793/0 (socket says 172.21.15.88:57793) 2024-03-29T10:43:32.412 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:32.426+0000 7faf35f54640 10 monclient(hunting): picked mon.noname-a con 0x7faf301457f0 addr v1:172.21.15.88:6789/0 2024-03-29T10:43:32.412 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:32.426+0000 7faf35f54640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf30141f00 con 0x7faf301457f0 2024-03-29T10:43:32.412 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:32.426+0000 7faf35f54640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf30142bd0 con 0x7faf3013f650 2024-03-29T10:43:32.412 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:32.426+0000 7faf35f54640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf30142120 con 0x7faf300a9d70 2024-03-29T10:43:32.412 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:32.426+0000 7faf35f54640 10 monclient(hunting): _renew_subs 2024-03-29T10:43:32.412 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:32.426+0000 7faf35f54640 10 monclient(hunting): authenticate will time out at 934.662312s 2024-03-29T10:43:32.412 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:32.426+0000 7faf2f7fe640 1 -- 172.21.15.88:0/3251993349 learned_addr learned my addr 172.21.15.88:0/3251993349 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-29T10:43:32.412 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:32.426+0000 7faf17fff640 10 client.?.objecter ms_handle_connect 0x7faf301457f0 2024-03-29T10:43:32.412 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:32.426+0000 7faf17fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:43:32.413 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:32.426+0000 7faf17fff640 1 -- 172.21.15.88:0/3251993349 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 947626476 0 0) 0x7faf30141f00 con 0x7faf301457f0 2024-03-29T10:43:32.413 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:32.426+0000 7faf17fff640 10 monclient(hunting): _negotiate 2024-03-29T10:43:32.413 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:32.426+0000 7faf17fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:43:32.413 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:32.426+0000 7faf17fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:43:32.413 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:32.427+0000 7faf17fff640 10 monclient(hunting): my global_id is 4320 2024-03-29T10:43:32.413 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:32.427+0000 7faf17fff640 1 -- 172.21.15.88:0/3251993349 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faf1c004eb0 con 0x7faf301457f0 2024-03-29T10:43:32.413 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:32.427+0000 7faf17fff640 10 client.?.objecter ms_handle_connect 0x7faf300a9d70 2024-03-29T10:43:32.413 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:32.427+0000 7faf17fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:43:32.413 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:32.427+0000 7faf17fff640 1 -- 172.21.15.88:0/3251993349 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3130157083 0 0) 0x7faf1c004eb0 con 0x7faf301457f0 2024-03-29T10:43:32.413 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:32.427+0000 7faf17fff640 1 -- 172.21.15.88:0/3251993349 --> v1:172.21.15.88:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7faf30141f00 con 0x7faf301457f0 2024-03-29T10:43:32.413 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:32.427+0000 7faf17fff640 1 -- 172.21.15.88:0/3251993349 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7faf20003f40 con 0x7faf301457f0 2024-03-29T10:43:32.413 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:32.427+0000 7faf17fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:43:32.413 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:32.427+0000 7faf17fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:43:32.413 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:32.427+0000 7faf17fff640 10 monclient(hunting): dump: 2024-03-29T10:43:32.413 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:43:32.414 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:43:32.414 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:43:32.414 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:43:32.414 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:43:32.414 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:43:32.414 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:43:32.414 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:43:32.414 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:43:32.414 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:43:32.414 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:32.427+0000 7faf17fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:43:32.414 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:32.427+0000 7faf17fff640 1 -- 172.21.15.88:0/3251993349 <== mon.0 v1:172.21.15.88:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 40111186 0 0) 0x7faf30141f00 con 0x7faf301457f0 2024-03-29T10:43:32.414 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:32.427+0000 7faf17fff640 1 -- 172.21.15.88:0/3251993349 >> v1:172.21.15.88:6790/0 conn(0x7faf300a9d70 legacy=0x7faf30136f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:32.414 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:32.428+0000 7faf17fff640 1 -- 172.21.15.88:0/3251993349 >> v1:172.21.15.196:6789/0 conn(0x7faf3013f650 legacy=0x7faf301d63c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:43:32.418 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:32.430+0000 7faf17fff640 10 monclient: _finish_hunting 0 2024-03-29T10:43:32.418 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:32.430+0000 7faf17fff640 1 monclient: found mon.a 2024-03-29T10:43:32.418 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:32.430+0000 7faf17fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:43:32.418 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:32.430+0000 7faf17fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-03-29T10:43:32.418 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:32.430+0000 7faf17fff640 1 -- 172.21.15.88:0/3251993349 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf301422b0 con 0x7faf301457f0 2024-03-29T10:43:32.418 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:32.430+0000 7faf17fff640 10 monclient: _finish_auth 0 2024-03-29T10:43:32.418 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:32.430+0000 7faf17fff640 10 monclient: _check_auth_tickets 2024-03-29T10:43:32.418 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:32.430+0000 7faf17fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:43:32.419 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:32.430+0000 7faf35f54640 5 monclient: authenticate success, global_id 4320 2024-03-29T10:43:32.419 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:32.430+0000 7faf35f54640 10 monclient: _renew_subs 2024-03-29T10:43:32.419 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:32.430+0000 7faf35f54640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-03-29T10:43:32.419 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:32.430+0000 7faf35f54640 1 -- 172.21.15.88:0/3251993349 --> v1:172.21.15.88:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf30137670 con 0x7faf301457f0 2024-03-29T10:43:32.419 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:32.430+0000 7faf35f54640 10 client.4320.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:43:32.419 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:32.430+0000 7faf35f54640 10 monclient: _renew_subs 2024-03-29T10:43:32.419 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:32.430+0000 7faf35f54640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-03-29T10:43:32.419 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:32.430+0000 7faf35f54640 1 -- 172.21.15.88:0/3251993349 --> v1:172.21.15.88:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7faf30139ae0 con 0x7faf301457f0 2024-03-29T10:43:32.419 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:32.430+0000 7faf35f54640 1 librados: init done 2024-03-29T10:43:32.419 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:32.430+0000 7faf17fff640 1 -- 172.21.15.88:0/3251993349 <== mon.0 v1:172.21.15.88:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7faf200044d0 con 0x7faf301457f0 2024-03-29T10:43:32.419 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:32.430+0000 7faf17fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:43:32.420 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:32.430+0000 7faf17fff640 1 -- 172.21.15.88:0/3251993349 <== mon.0 v1:172.21.15.88:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7faf20000ed0 con 0x7faf301457f0 2024-03-29T10:43:32.420 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:32.430+0000 7faf17fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:43:32.420 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:32.430+0000 7faf17fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:43:32.420 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:32.430+0000 7faf17fff640 10 monclient: dump: 2024-03-29T10:43:32.420 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:43:32.420 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:43:32.420 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:43:32.420 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:43:32.420 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:43:32.420 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:43:32.420 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:43:32.420 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:43:32.421 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:43:32.421 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:43:32.421 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:32.430+0000 7faf2dffb640 4 set_mon_vals no callback set 2024-03-29T10:43:32.421 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:32.432+0000 7faf17fff640 1 -- 172.21.15.88:0/3251993349 <== mon.0 v1:172.21.15.88:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4235629220 0 0) 0x7faf20001030 con 0x7faf301457f0 2024-03-29T10:43:32.421 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:32.433+0000 7faf17fff640 1 -- 172.21.15.88:0/3251993349 <== mon.0 v1:172.21.15.88:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4257673582 0 0) 0x7faf200847c0 con 0x7faf301457f0 2024-03-29T10:43:32.421 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:32.433+0000 7faf17fff640 10 client.4320.objecter ms_dispatch 0x7faf30000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:43:32.421 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:32.433+0000 7faf17fff640 3 client.4320.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:43:32.421 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:32.433+0000 7faf17fff640 3 client.4320.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:43:32.421 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:32.433+0000 7faf17fff640 20 client.4320.objecter dump_active .. 0 homeless 2024-03-29T10:43:32.421 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:32.433+0000 7faf35f54640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:43:32.422 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:32.434+0000 7faf35f54640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:43:32.422 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:32.434+0000 7faf35f54640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-03-29T10:43:32.422 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:32.434+0000 7faf35f54640 1 -- 172.21.15.88:0/3251993349 --> v1:172.21.15.88:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf10005080 con 0x7faf301457f0 2024-03-29T10:43:32.422 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:32.434+0000 7faf2f7fe640 10 monclient: get_auth_request con 0x7faf1c06dbf0 auth_method 0 2024-03-29T10:43:32.422 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:32.434+0000 7faf17fff640 10 client.4320.objecter ms_handle_connect 0x7faf1c06dbf0 2024-03-29T10:43:32.428 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:32.445+0000 7faf17fff640 1 -- 172.21.15.88:0/3251993349 <== mon.0 v1:172.21.15.88:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7faf200500a0 con 0x7faf301457f0 2024-03-29T10:43:32.428 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:32.445+0000 7faf17fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:43:32.428 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:32.445+0000 7faf17fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:43:32.741 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:32.758+0000 7faf35f54640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:43:32.741 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:32.758+0000 7faf35f54640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:43:32.741 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:32.758+0000 7faf35f54640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-03-29T10:43:32.741 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:32.758+0000 7faf35f54640 1 -- 172.21.15.88:0/3251993349 --> v1:172.21.15.88:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7faf10005330 con 0x7faf301457f0 2024-03-29T10:43:32.742 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:32.759+0000 7faf17fff640 1 -- 172.21.15.88:0/3251993349 <== mon.0 v1:172.21.15.88:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 344506374) 0x7faf20051fd0 con 0x7faf301457f0 2024-03-29T10:43:32.742 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:32.759+0000 7faf17fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:43:32.742 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:32.759+0000 7faf17fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:43:32.744 INFO:teuthology.misc.health.smithi088.stdout: 2024-03-29T10:43:32.744 INFO:teuthology.misc.health.smithi088.stdout:{"epoch":10,"fsid":"869f5188-f92b-42dd-a46d-8c646e0388a9","created":"2024-03-29T10:40:40.285087+0000","modified":"2024-03-29T10:40:42.536230+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:40:42.536230+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a26c0c1c-8de0-438c-a3f6-3df10ea61574","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5de24915-8dec-49b6-8b16-f3170197c591","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ac054af3-d589-4565-9644-e6af75975c53","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3ec4f045-c714-4a22-bd1a-e07330b45ab3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"cddb9867-8e30-4951-9b13-3d1e78b9c45a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dc6d9911-785d-432f-8b53-56a1ccab302f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"77c88437-b587-496e-8b4d-1824ea8b5305","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ce963d96-5a8e-44c3-99de-c0dbebe234ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:43:32.747 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:32.764+0000 7faf35f54640 10 librados: watch_flush enter 2024-03-29T10:43:32.750 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:32.768+0000 7faf35f54640 10 librados: watch_flush exit 2024-03-29T10:43:32.751 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:32.768+0000 7faf35f54640 20 client.4320.objecter shutdown clearing up homeless session... 2024-03-29T10:43:32.751 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:32.768+0000 7faf35f54640 10 client.4320.objecter successfully canceled tick 2024-03-29T10:43:32.757 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:32.774+0000 7faf35f54640 1 -- 172.21.15.88:0/3251993349 >> v1:172.21.15.88:6816/1752861349 conn(0x7faf1c06dbf0 legacy=0x7faf1c0700b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:32.757 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:32.774+0000 7faf35f54640 10 monclient: shutdown 2024-03-29T10:43:32.758 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:32.774+0000 7faf35f54640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:43:32.758 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:32.774+0000 7faf35f54640 1 -- 172.21.15.88:0/3251993349 >> v1:172.21.15.88:6789/0 conn(0x7faf301457f0 legacy=0x7faf301d6ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:32.758 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:32.774+0000 7faf35f54640 1 -- 172.21.15.88:0/3251993349 shutdown_connections 2024-03-29T10:43:32.758 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:32.774+0000 7faf35f54640 1 -- 172.21.15.88:0/3251993349 >> 172.21.15.88:0/3251993349 conn(0x7faf30066a00 msgr2=0x7faf3013a6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:43:32.758 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:32.774+0000 7faf35f54640 1 -- 172.21.15.88:0/3251993349 shutdown_connections 2024-03-29T10:43:32.758 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:32.774+0000 7faf35f54640 1 -- 172.21.15.88:0/3251993349 wait complete. 2024-03-29T10:43:32.759 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:32.774+0000 7faf35f54640 1 librados: shutdown 2024-03-29T10:43:32.774 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:43:39.580 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:43:39.728 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:39.745+0000 7f51f5c9d640 10 monclient: get_monmap_and_config 2024-03-29T10:43:39.728 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:39.745+0000 7f51f5c9d640 10 monclient: build_initial_monmap 2024-03-29T10:43:39.728 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:39.745+0000 7f51f5c9d640 10 monclient: monmap: 2024-03-29T10:43:39.729 INFO:teuthology.misc.health.smithi088.stderr:epoch 0 2024-03-29T10:43:39.729 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:43:39.729 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:43:39.746725+0000 2024-03-29T10:43:39.729 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:43:39.746725+0000 2024-03-29T10:43:39.729 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 0 (unknown) 2024-03-29T10:43:39.729 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:43:39.729 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-03-29T10:43:39.729 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-29T10:43:39.729 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-b 2024-03-29T10:43:39.729 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:43:39.731 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:39.748+0000 7f51f5c9d640 1 Processor -- start 2024-03-29T10:43:39.732 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:39.749+0000 7f51f5c9d640 1 -- start start 2024-03-29T10:43:39.732 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:39.749+0000 7f51f5c9d640 10 monclient: init 2024-03-29T10:43:39.733 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:39.750+0000 7f51f5c9d640 10 monclient: _reopen_session rank -1 2024-03-29T10:43:39.733 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:39.750+0000 7f51f5c9d640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:43:39.734 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:39.750+0000 7f51f5c9d640 10 monclient(hunting): picked mon.noname-b con 0x7f51e80389e0 addr v1:172.21.15.88:6790/0 2024-03-29T10:43:39.734 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:39.750+0000 7f51f5c9d640 10 monclient(hunting): picked mon.noname-a con 0x7f51e8038dc0 addr v1:172.21.15.88:6789/0 2024-03-29T10:43:39.734 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:39.750+0000 7f51f5c9d640 10 monclient(hunting): picked mon.noname-c con 0x7f51e8030660 addr v1:172.21.15.196:6789/0 2024-03-29T10:43:39.734 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:39.750+0000 7f51f5c9d640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51e80c54c0 con 0x7f51e8038dc0 2024-03-29T10:43:39.734 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:39.750+0000 7f51f5c9d640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51e8036430 con 0x7f51e8030660 2024-03-29T10:43:39.734 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:39.750+0000 7f51f5c9d640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51e8039840 con 0x7f51e80389e0 2024-03-29T10:43:39.735 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:39.750+0000 7f51f5c9d640 10 monclient(hunting): _renew_subs 2024-03-29T10:43:39.735 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:39.750+0000 7f51f5c9d640 10 monclient(hunting): authenticate will time out at 941.986794s 2024-03-29T10:43:39.738 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:39.754+0000 7f51f549c640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f51e8030660 0x7f51e80d3170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.88:56679/0 (socket says 172.21.15.88:56679) 2024-03-29T10:43:39.738 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:39.755+0000 7f51f549c640 1 -- 172.21.15.88:0/1977273933 learned_addr learned my addr 172.21.15.88:0/1977273933 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-29T10:43:39.742 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:39.756+0000 7f51ef7fe640 1 -- 172.21.15.88:0/1977273933 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1494505910 0 0) 0x7f51e8036430 con 0x7f51e8030660 2024-03-29T10:43:39.742 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:39.756+0000 7f51ef7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:43:39.742 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:39.756+0000 7f51ef7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:43:39.742 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:39.756+0000 7f51ef7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:43:39.742 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:39.756+0000 7f51ef7fe640 10 monclient(hunting): my global_id is 4318 2024-03-29T10:43:39.742 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:39.757+0000 7f51ef7fe640 1 -- 172.21.15.88:0/1977273933 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f51e0003520 con 0x7f51e8030660 2024-03-29T10:43:39.742 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:39.757+0000 7f51ef7fe640 1 -- 172.21.15.88:0/1977273933 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4129720480 0 0) 0x7f51e8039840 con 0x7f51e80389e0 2024-03-29T10:43:39.742 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:39.757+0000 7f51ef7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:43:39.743 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:39.757+0000 7f51ef7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:43:39.743 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:39.757+0000 7f51ef7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:43:39.743 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:39.757+0000 7f51ef7fe640 10 monclient(hunting): my global_id is 4349 2024-03-29T10:43:39.743 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:39.757+0000 7f51ef7fe640 1 -- 172.21.15.88:0/1977273933 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f51e8036430 con 0x7f51e80389e0 2024-03-29T10:43:39.743 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:39.757+0000 7f51ef7fe640 1 -- 172.21.15.88:0/1977273933 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 740264956 0 0) 0x7f51e0003520 con 0x7f51e8030660 2024-03-29T10:43:39.743 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:39.757+0000 7f51ef7fe640 1 -- 172.21.15.88:0/1977273933 >> v1:172.21.15.88:6790/0 conn(0x7f51e80389e0 legacy=0x7f51e8033520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:39.743 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:39.757+0000 7f51ef7fe640 1 -- 172.21.15.88:0/1977273933 >> v1:172.21.15.88:6789/0 conn(0x7f51e8038dc0 legacy=0x7f51e80cadd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:39.743 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:39.757+0000 7f51ef7fe640 10 monclient: _finish_hunting 0 2024-03-29T10:43:39.743 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:39.757+0000 7f51ef7fe640 1 monclient: found mon.noname-c 2024-03-29T10:43:39.743 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:39.757+0000 7f51ef7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:43:39.743 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:39.758+0000 7f51ef7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.196:6789/0 2024-03-29T10:43:39.743 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:39.758+0000 7f51ef7fe640 1 -- 172.21.15.88:0/1977273933 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51e8035690 con 0x7f51e8030660 2024-03-29T10:43:39.743 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:39.758+0000 7f51ef7fe640 10 monclient: _finish_auth 0 2024-03-29T10:43:39.743 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:39.758+0000 7f51ef7fe640 10 monclient: _check_auth_tickets 2024-03-29T10:43:39.743 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:39.758+0000 7f51ef7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:43:39.744 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:39.758+0000 7f51ef7fe640 1 -- 172.21.15.88:0/1977273933 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f51e4001690 con 0x7f51e8030660 2024-03-29T10:43:39.744 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:39.758+0000 7f51ef7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:43:39.744 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:39.758+0000 7f51ef7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:43:39.744 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:39.758+0000 7f51ef7fe640 10 monclient: dump: 2024-03-29T10:43:39.744 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:43:39.744 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:43:39.744 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:43:39.744 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:43:39.744 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:43:39.744 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:43:39.744 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:43:39.744 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:43:39.744 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:43:39.744 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:43:39.745 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:39.758+0000 7f51f5c9d640 5 monclient: authenticate success, global_id 4318 2024-03-29T10:43:39.745 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:39.758+0000 7f51f5c9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:43:39.745 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:39.759+0000 7f51ef7fe640 1 -- 172.21.15.88:0/1977273933 <== mon.1 v1:172.21.15.196:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f51e4001920 con 0x7f51e8030660 2024-03-29T10:43:39.745 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:39.759+0000 7f51ef7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:43:39.745 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:39.759+0000 7f51ef7fe640 1 -- 172.21.15.88:0/1977273933 <== mon.1 v1:172.21.15.196:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f51e4001cb0 con 0x7f51e8030660 2024-03-29T10:43:39.745 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:39.759+0000 7f51ef7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:43:39.745 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:39.759+0000 7f51ef7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:43:39.745 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:39.759+0000 7f51ef7fe640 10 monclient: dump: 2024-03-29T10:43:39.746 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:43:39.746 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:43:39.746 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:43:39.746 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:43:39.746 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:43:39.746 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:43:39.746 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:43:39.746 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:43:39.746 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:43:39.746 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:43:39.746 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:39.759+0000 7f51f5c9d640 10 monclient: get_monmap_and_config success 2024-03-29T10:43:39.746 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:39.759+0000 7f51f5c9d640 4 set_mon_vals no callback set 2024-03-29T10:43:39.746 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:39.759+0000 7f51f5c9d640 10 monclient: shutdown 2024-03-29T10:43:39.746 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:39.759+0000 7f51f5c9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:43:39.747 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:39.759+0000 7f51f5c9d640 1 -- 172.21.15.88:0/1977273933 >> v1:172.21.15.196:6789/0 conn(0x7f51e8030660 legacy=0x7f51e80d3170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:39.747 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:39.760+0000 7f51f5c9d640 1 -- 172.21.15.88:0/1977273933 shutdown_connections 2024-03-29T10:43:39.747 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:39.760+0000 7f51f5c9d640 1 -- 172.21.15.88:0/1977273933 >> 172.21.15.88:0/1977273933 conn(0x7f51e8005c00 msgr2=0x7f51e8006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:43:39.747 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:39.764+0000 7f51f5c9d640 1 -- 172.21.15.88:0/1977273933 shutdown_connections 2024-03-29T10:43:39.748 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:39.764+0000 7f51f5c9d640 1 -- 172.21.15.88:0/1977273933 wait complete. 2024-03-29T10:43:39.749 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:39.765+0000 7f51f5c9d640 10 monclient: build_initial_monmap 2024-03-29T10:43:39.749 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:39.765+0000 7f51f5c9d640 10 monclient: monmap: 2024-03-29T10:43:39.750 INFO:teuthology.misc.health.smithi088.stderr:epoch 0 2024-03-29T10:43:39.750 INFO:teuthology.misc.health.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:43:39.750 INFO:teuthology.misc.health.smithi088.stderr:last_changed 0.000000 2024-03-29T10:43:39.750 INFO:teuthology.misc.health.smithi088.stderr:created 0.000000 2024-03-29T10:43:39.750 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 0 (unknown) 2024-03-29T10:43:39.750 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:43:39.750 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-03-29T10:43:39.750 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-29T10:43:39.750 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-c 2024-03-29T10:43:39.750 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:43:39.750 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:39.766+0000 7f51f5c9d640 1 librados: starting msgr at 2024-03-29T10:43:39.750 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:39.766+0000 7f51f5c9d640 1 librados: starting objecter 2024-03-29T10:43:39.750 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:39.766+0000 7f51f5c9d640 1 Processor -- start 2024-03-29T10:43:39.751 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:39.766+0000 7f51f5c9d640 1 -- start start 2024-03-29T10:43:39.751 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:39.766+0000 7f51f5c9d640 1 librados: setting wanted keys 2024-03-29T10:43:39.751 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:39.766+0000 7f51f5c9d640 1 librados: calling monclient init 2024-03-29T10:43:39.751 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:39.766+0000 7f51f5c9d640 10 monclient: init 2024-03-29T10:43:39.752 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:39.768+0000 7f51f5c9d640 10 monclient: _reopen_session rank -1 2024-03-29T10:43:39.752 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:39.768+0000 7f51f5c9d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:43:39.752 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:39.768+0000 7f51f5c9d640 10 monclient(hunting): picked mon.noname-a con 0x7f51e8030660 addr v1:172.21.15.88:6789/0 2024-03-29T10:43:39.752 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:39.768+0000 7f51f5c9d640 10 monclient(hunting): picked mon.noname-b con 0x7f51e80389e0 addr v1:172.21.15.196:6789/0 2024-03-29T10:43:39.752 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:39.768+0000 7f51f5c9d640 10 monclient(hunting): picked mon.noname-c con 0x7f51e80c5650 addr v1:172.21.15.88:6790/0 2024-03-29T10:43:39.752 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:39.768+0000 7f51f5c9d640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51e815a440 con 0x7f51e8030660 2024-03-29T10:43:39.752 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:39.768+0000 7f51f5c9d640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51e815cfa0 con 0x7f51e80389e0 2024-03-29T10:43:39.753 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:39.768+0000 7f51f5c9d640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51e80bfb50 con 0x7f51e80c5650 2024-03-29T10:43:39.753 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:39.768+0000 7f51f5c9d640 10 monclient(hunting): _renew_subs 2024-03-29T10:43:39.753 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:39.768+0000 7f51f5c9d640 10 monclient(hunting): authenticate will time out at 942.004109s 2024-03-29T10:43:39.758 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:39.773+0000 7f51f549c640 1 --1- >> v1:172.21.15.88:6790/0 conn(0x7f51e80c5650 0x7f51e81681b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6790/0 says I am v1:172.21.15.88:45515/0 (socket says 172.21.15.88:45515) 2024-03-29T10:43:39.758 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:39.773+0000 7f51f549c640 1 -- 172.21.15.88:0/1361141776 learned_addr learned my addr 172.21.15.88:0/1361141776 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-29T10:43:39.758 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:39.773+0000 7f51ed7fa640 10 client.?.objecter ms_handle_connect 0x7f51e80c5650 2024-03-29T10:43:39.758 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:39.773+0000 7f51ed7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:43:39.758 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:39.773+0000 7f51ed7fa640 1 -- 172.21.15.88:0/1361141776 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2936873724 0 0) 0x7f51e80bfb50 con 0x7f51e80c5650 2024-03-29T10:43:39.758 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:39.773+0000 7f51ed7fa640 10 monclient(hunting): _negotiate 2024-03-29T10:43:39.758 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:39.773+0000 7f51ed7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:43:39.758 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:39.773+0000 7f51ed7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:43:39.758 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:39.773+0000 7f51ed7fa640 10 monclient(hunting): my global_id is 4352 2024-03-29T10:43:39.758 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:39.773+0000 7f51ed7fa640 1 -- 172.21.15.88:0/1361141776 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f51d4004eb0 con 0x7f51e80c5650 2024-03-29T10:43:39.758 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:39.773+0000 7f51ed7fa640 10 client.?.objecter ms_handle_connect 0x7f51e8030660 2024-03-29T10:43:39.759 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:39.773+0000 7f51ed7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:43:39.759 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:39.773+0000 7f51ed7fa640 10 client.?.objecter ms_handle_connect 0x7f51e80389e0 2024-03-29T10:43:39.759 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:39.774+0000 7f51ed7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:43:39.759 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:39.774+0000 7f51ed7fa640 1 -- 172.21.15.88:0/1361141776 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3957573743 0 0) 0x7f51d4004eb0 con 0x7f51e80c5650 2024-03-29T10:43:39.759 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:39.774+0000 7f51ed7fa640 1 -- 172.21.15.88:0/1361141776 --> v1:172.21.15.88:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f51e80bfb50 con 0x7f51e80c5650 2024-03-29T10:43:39.759 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:39.774+0000 7f51ed7fa640 1 -- 172.21.15.88:0/1361141776 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f51e4001550 con 0x7f51e80c5650 2024-03-29T10:43:39.759 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:39.774+0000 7f51ed7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:43:39.759 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:39.774+0000 7f51ed7fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:43:39.759 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:39.774+0000 7f51ed7fa640 10 monclient(hunting): dump: 2024-03-29T10:43:39.759 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:43:39.759 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:43:39.759 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:43:39.759 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:43:39.759 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:43:39.759 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:43:39.760 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:43:39.760 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:43:39.760 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:43:39.760 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:43:39.760 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:39.777+0000 7f51ed7fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:43:39.760 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:39.777+0000 7f51ed7fa640 1 -- 172.21.15.88:0/1361141776 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1024201632 0 0) 0x7f51e815a440 con 0x7f51e8030660 2024-03-29T10:43:39.760 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:39.777+0000 7f51ed7fa640 10 monclient(hunting): _negotiate 2024-03-29T10:43:39.761 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:39.777+0000 7f51ed7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:43:39.761 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:39.777+0000 7f51ed7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:43:39.761 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:39.777+0000 7f51ed7fa640 10 monclient(hunting): my global_id is 4323 2024-03-29T10:43:39.761 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:39.777+0000 7f51ed7fa640 1 -- 172.21.15.88:0/1361141776 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f51d4004eb0 con 0x7f51e8030660 2024-03-29T10:43:39.761 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:39.777+0000 7f51ed7fa640 1 -- 172.21.15.88:0/1361141776 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2531592600 0 0) 0x7f51e815cfa0 con 0x7f51e80389e0 2024-03-29T10:43:39.761 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:39.777+0000 7f51ed7fa640 10 monclient(hunting): _negotiate 2024-03-29T10:43:39.761 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:39.777+0000 7f51ed7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:43:39.761 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:39.777+0000 7f51ed7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:43:39.761 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:39.777+0000 7f51ed7fa640 10 monclient(hunting): my global_id is 4321 2024-03-29T10:43:39.761 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:39.777+0000 7f51ed7fa640 1 -- 172.21.15.88:0/1361141776 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f51e815a440 con 0x7f51e80389e0 2024-03-29T10:43:39.761 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:39.777+0000 7f51ed7fa640 1 -- 172.21.15.88:0/1361141776 <== mon.2 v1:172.21.15.88:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1497174456 0 0) 0x7f51e80bfb50 con 0x7f51e80c5650 2024-03-29T10:43:39.761 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:39.777+0000 7f51ed7fa640 1 -- 172.21.15.88:0/1361141776 >> v1:172.21.15.196:6789/0 conn(0x7f51e80389e0 legacy=0x7f51e80bf440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:39.761 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:39.777+0000 7f51ed7fa640 1 -- 172.21.15.88:0/1361141776 >> v1:172.21.15.88:6789/0 conn(0x7f51e8030660 legacy=0x7f51e815c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:39.762 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:39.777+0000 7f51ed7fa640 10 monclient: _finish_hunting 0 2024-03-29T10:43:39.763 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:39.777+0000 7f51ed7fa640 1 monclient: found mon.c 2024-03-29T10:43:39.763 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:39.777+0000 7f51ed7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:43:39.763 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:39.777+0000 7f51ed7fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-03-29T10:43:39.763 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:39.777+0000 7f51ed7fa640 1 -- 172.21.15.88:0/1361141776 --> v1:172.21.15.88:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51e80bfce0 con 0x7f51e80c5650 2024-03-29T10:43:39.763 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:39.778+0000 7f51ed7fa640 10 monclient: _finish_auth 0 2024-03-29T10:43:39.763 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:39.778+0000 7f51ed7fa640 10 monclient: _check_auth_tickets 2024-03-29T10:43:39.763 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:39.778+0000 7f51ed7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:43:39.763 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:39.778+0000 7f51f5c9d640 5 monclient: authenticate success, global_id 4352 2024-03-29T10:43:39.763 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:39.778+0000 7f51f5c9d640 10 monclient: _renew_subs 2024-03-29T10:43:39.763 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:39.778+0000 7f51f5c9d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-03-29T10:43:39.763 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:39.778+0000 7f51f5c9d640 1 -- 172.21.15.88:0/1361141776 --> v1:172.21.15.88:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51e802f000 con 0x7f51e80c5650 2024-03-29T10:43:39.763 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:39.778+0000 7f51f5c9d640 10 client.4352.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:43:39.763 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:39.778+0000 7f51f5c9d640 10 monclient: _renew_subs 2024-03-29T10:43:39.764 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:39.778+0000 7f51f5c9d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-03-29T10:43:39.764 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:39.778+0000 7f51f5c9d640 1 -- 172.21.15.88:0/1361141776 --> v1:172.21.15.88:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f51e80ca310 con 0x7f51e80c5650 2024-03-29T10:43:39.764 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:39.778+0000 7f51f5c9d640 1 librados: init done 2024-03-29T10:43:39.764 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:39.779+0000 7f51daffd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:43:39.764 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:39.779+0000 7f51daffd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:43:39.764 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:39.779+0000 7f51daffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-03-29T10:43:39.764 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:39.779+0000 7f51daffd640 1 -- 172.21.15.88:0/1361141776 --> v1:172.21.15.88:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51e802ee50 con 0x7f51e80c5650 2024-03-29T10:43:39.766 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:39.783+0000 7f51ed7fa640 1 -- 172.21.15.88:0/1361141776 <== mon.2 v1:172.21.15.88:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f51e40016b0 con 0x7f51e80c5650 2024-03-29T10:43:39.766 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:39.783+0000 7f51ed7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:43:39.767 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:39.783+0000 7f51ed7fa640 1 -- 172.21.15.88:0/1361141776 <== mon.2 v1:172.21.15.88:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f51e4004d90 con 0x7f51e80c5650 2024-03-29T10:43:39.767 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:39.783+0000 7f51ed7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:43:39.767 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:39.783+0000 7f51ed7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:43:39.767 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:39.783+0000 7f51eeffd640 4 set_mon_vals no callback set 2024-03-29T10:43:39.767 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:39.783+0000 7f51ed7fa640 10 monclient: dump: 2024-03-29T10:43:39.767 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:43:39.767 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:43:39.767 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:43:39.767 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:43:39.767 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:43:39.767 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:43:39.767 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:43:39.767 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:43:39.767 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:43:39.768 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:43:39.768 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:39.783+0000 7f51ed7fa640 1 -- 172.21.15.88:0/1361141776 <== mon.2 v1:172.21.15.88:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4235629220 0 0) 0x7f51e401ad60 con 0x7f51e80c5650 2024-03-29T10:43:39.768 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:39.784+0000 7f51f4c9b640 10 monclient: get_auth_request con 0x7f51d406de80 auth_method 0 2024-03-29T10:43:39.769 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:39.785+0000 7f51ed7fa640 1 -- 172.21.15.88:0/1361141776 <== mon.2 v1:172.21.15.88:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4257673582 0 0) 0x7f51e4084a50 con 0x7f51e80c5650 2024-03-29T10:43:39.769 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:39.785+0000 7f51ed7fa640 10 client.4352.objecter ms_dispatch 0x7f51f0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:43:39.769 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:39.785+0000 7f51ed7fa640 3 client.4352.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:43:39.769 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:39.785+0000 7f51ed7fa640 3 client.4352.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:43:39.769 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:39.785+0000 7f51ed7fa640 20 client.4352.objecter dump_active .. 0 homeless 2024-03-29T10:43:39.769 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:39.785+0000 7f51ed7fa640 10 client.4352.objecter ms_handle_connect 0x7f51d406de80 2024-03-29T10:43:39.775 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:39.792+0000 7f51ed7fa640 1 -- 172.21.15.88:0/1361141776 <== mon.2 v1:172.21.15.88:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f51e4050490 con 0x7f51e80c5650 2024-03-29T10:43:39.775 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:39.792+0000 7f51ed7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:43:39.775 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:39.792+0000 7f51ed7fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:43:40.106 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:40.121+0000 7f51daffd640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:43:40.107 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:40.121+0000 7f51daffd640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:43:40.107 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:40.121+0000 7f51daffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-03-29T10:43:40.107 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:40.121+0000 7f51daffd640 1 -- 172.21.15.88:0/1361141776 --> v1:172.21.15.88:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f51e8007b90 con 0x7f51e80c5650 2024-03-29T10:43:40.107 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:40.123+0000 7f51ed7fa640 1 -- 172.21.15.88:0/1361141776 <== mon.2 v1:172.21.15.88:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 344506374) 0x7f51e40523c0 con 0x7f51e80c5650 2024-03-29T10:43:40.107 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:40.123+0000 7f51ed7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:43:40.107 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:40.123+0000 7f51ed7fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:43:40.107 INFO:teuthology.misc.health.smithi088.stdout: 2024-03-29T10:43:40.108 INFO:teuthology.misc.health.smithi088.stdout:{"epoch":10,"fsid":"869f5188-f92b-42dd-a46d-8c646e0388a9","created":"2024-03-29T10:40:40.285087+0000","modified":"2024-03-29T10:40:42.536230+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:40:42.536230+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a26c0c1c-8de0-438c-a3f6-3df10ea61574","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5de24915-8dec-49b6-8b16-f3170197c591","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ac054af3-d589-4565-9644-e6af75975c53","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3ec4f045-c714-4a22-bd1a-e07330b45ab3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"cddb9867-8e30-4951-9b13-3d1e78b9c45a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dc6d9911-785d-432f-8b53-56a1ccab302f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"77c88437-b587-496e-8b4d-1824ea8b5305","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ce963d96-5a8e-44c3-99de-c0dbebe234ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:43:40.110 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:40.127+0000 7f51daffd640 10 librados: watch_flush enter 2024-03-29T10:43:40.111 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:40.128+0000 7f51daffd640 10 librados: watch_flush exit 2024-03-29T10:43:40.111 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:40.128+0000 7f51daffd640 20 client.4352.objecter shutdown clearing up homeless session... 2024-03-29T10:43:40.111 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:40.128+0000 7f51daffd640 10 client.4352.objecter successfully canceled tick 2024-03-29T10:43:40.111 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:40.128+0000 7f51daffd640 1 -- 172.21.15.88:0/1361141776 >> v1:172.21.15.88:6816/1752861349 conn(0x7f51d406de80 legacy=0x7f51d4070340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:40.112 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:40.128+0000 7f51daffd640 10 monclient: shutdown 2024-03-29T10:43:40.112 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:40.128+0000 7f51daffd640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:43:40.112 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:40.128+0000 7f51daffd640 1 -- 172.21.15.88:0/1361141776 >> v1:172.21.15.88:6790/0 conn(0x7f51e80c5650 legacy=0x7f51e81681b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:40.112 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:40.128+0000 7f51daffd640 1 -- 172.21.15.88:0/1361141776 shutdown_connections 2024-03-29T10:43:40.112 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:40.128+0000 7f51daffd640 1 -- 172.21.15.88:0/1361141776 >> 172.21.15.88:0/1361141776 conn(0x7f51e8005c00 msgr2=0x7f51e80c3800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:43:40.112 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:40.128+0000 7f51daffd640 1 -- 172.21.15.88:0/1361141776 shutdown_connections 2024-03-29T10:43:40.112 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:40.128+0000 7f51daffd640 1 -- 172.21.15.88:0/1361141776 wait complete. 2024-03-29T10:43:40.114 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:40.131+0000 7f51daffd640 1 librados: shutdown 2024-03-29T10:43:40.131 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:43:46.939 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:43:47.090 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:47.107+0000 7f56c6c12640 10 monclient: get_monmap_and_config 2024-03-29T10:43:47.090 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:47.107+0000 7f56c6c12640 10 monclient: build_initial_monmap 2024-03-29T10:43:47.091 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:47.107+0000 7f56c6c12640 10 monclient: monmap: 2024-03-29T10:43:47.091 INFO:teuthology.misc.health.smithi088.stderr:epoch 0 2024-03-29T10:43:47.091 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:43:47.091 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:43:47.108127+0000 2024-03-29T10:43:47.091 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:43:47.108127+0000 2024-03-29T10:43:47.091 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 0 (unknown) 2024-03-29T10:43:47.092 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:43:47.092 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-03-29T10:43:47.092 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-29T10:43:47.092 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-b 2024-03-29T10:43:47.092 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:43:47.092 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:47.109+0000 7f56c6c12640 1 Processor -- start 2024-03-29T10:43:47.093 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:47.109+0000 7f56c6c12640 1 -- start start 2024-03-29T10:43:47.093 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:47.109+0000 7f56c6c12640 10 monclient: init 2024-03-29T10:43:47.093 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:47.110+0000 7f56c6c12640 10 monclient: _reopen_session rank -1 2024-03-29T10:43:47.093 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:47.110+0000 7f56c6c12640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:43:47.093 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:47.110+0000 7f56c6c12640 10 monclient(hunting): picked mon.noname-c con 0x7f56c00a9750 addr v1:172.21.15.196:6789/0 2024-03-29T10:43:47.093 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:47.110+0000 7f56c6c12640 10 monclient(hunting): picked mon.noname-a con 0x7f56c00add40 addr v1:172.21.15.88:6789/0 2024-03-29T10:43:47.096 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:47.110+0000 7f56c6c12640 10 monclient(hunting): picked mon.noname-b con 0x7f56c00aad20 addr v1:172.21.15.88:6790/0 2024-03-29T10:43:47.097 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:47.113+0000 7f56c6c12640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56c0149870 con 0x7f56c00add40 2024-03-29T10:43:47.097 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:47.113+0000 7f56c6c12640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56c00abf80 con 0x7f56c00a9750 2024-03-29T10:43:47.097 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:47.113+0000 7f56c6c12640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56c0149ac0 con 0x7f56c00aad20 2024-03-29T10:43:47.097 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:47.113+0000 7f56c6c12640 10 monclient(hunting): _renew_subs 2024-03-29T10:43:47.097 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:47.113+0000 7f56c6c12640 10 monclient(hunting): authenticate will time out at 949.349225s 2024-03-29T10:43:47.097 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:47.113+0000 7f56c6411640 1 --1- >> v1:172.21.15.88:6790/0 conn(0x7f56c00aad20 0x7f56c0151bf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6790/0 says I am v1:172.21.15.88:43303/0 (socket says 172.21.15.88:43303) 2024-03-29T10:43:47.097 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:47.113+0000 7f56c6411640 1 -- 172.21.15.88:0/2845016713 learned_addr learned my addr 172.21.15.88:0/2845016713 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-29T10:43:47.098 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:47.114+0000 7f56c4c0e640 1 -- 172.21.15.88:0/2845016713 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1643122498 0 0) 0x7f56c0149ac0 con 0x7f56c00aad20 2024-03-29T10:43:47.098 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:47.114+0000 7f56c4c0e640 10 monclient(hunting): _negotiate 2024-03-29T10:43:47.098 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:47.114+0000 7f56c4c0e640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:43:47.098 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:47.114+0000 7f56c4c0e640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:43:47.098 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:47.114+0000 7f56c4c0e640 10 monclient(hunting): my global_id is 4355 2024-03-29T10:43:47.098 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:47.114+0000 7f56c4c0e640 1 -- 172.21.15.88:0/2845016713 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f56ac003520 con 0x7f56c00aad20 2024-03-29T10:43:47.098 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:47.114+0000 7f56c4c0e640 1 -- 172.21.15.88:0/2845016713 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3905219187 0 0) 0x7f56ac003520 con 0x7f56c00aad20 2024-03-29T10:43:47.098 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:47.114+0000 7f56c4c0e640 1 -- 172.21.15.88:0/2845016713 >> v1:172.21.15.196:6789/0 conn(0x7f56c00a9750 legacy=0x7f56c0149fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:43:47.098 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:47.114+0000 7f56c4c0e640 1 -- 172.21.15.88:0/2845016713 >> v1:172.21.15.88:6789/0 conn(0x7f56c00add40 legacy=0x7f56c0143610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:43:47.098 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:47.114+0000 7f56c4c0e640 10 monclient: _finish_hunting 0 2024-03-29T10:43:47.098 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:47.114+0000 7f56c4c0e640 1 monclient: found mon.noname-b 2024-03-29T10:43:47.099 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:47.114+0000 7f56c4c0e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:43:47.099 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:47.114+0000 7f56c4c0e640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.88:6790/0 2024-03-29T10:43:47.099 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:47.114+0000 7f56c4c0e640 1 -- 172.21.15.88:0/2845016713 --> v1:172.21.15.88:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56c014a6f0 con 0x7f56c00aad20 2024-03-29T10:43:47.099 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:47.114+0000 7f56c4c0e640 10 monclient: _finish_auth 0 2024-03-29T10:43:47.099 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:47.114+0000 7f56c4c0e640 10 monclient: _check_auth_tickets 2024-03-29T10:43:47.099 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:47.114+0000 7f56c4c0e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:43:47.099 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:47.114+0000 7f56c4c0e640 1 -- 172.21.15.88:0/2845016713 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f56b0001690 con 0x7f56c00aad20 2024-03-29T10:43:47.099 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:47.114+0000 7f56c4c0e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:43:47.099 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:47.114+0000 7f56c4c0e640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:43:47.099 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:47.114+0000 7f56c4c0e640 10 monclient: dump: 2024-03-29T10:43:47.099 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:43:47.099 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:43:47.099 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:43:47.099 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:43:47.099 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:43:47.100 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:43:47.100 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:43:47.100 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:43:47.100 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:43:47.100 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:43:47.100 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:47.114+0000 7f56c6c12640 5 monclient: authenticate success, global_id 4355 2024-03-29T10:43:47.100 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:47.114+0000 7f56c6c12640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:43:47.100 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:47.115+0000 7f56c4c0e640 1 -- 172.21.15.88:0/2845016713 <== mon.2 v1:172.21.15.88:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f56b0001860 con 0x7f56c00aad20 2024-03-29T10:43:47.101 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:47.115+0000 7f56c4c0e640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:43:47.101 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:47.115+0000 7f56c4c0e640 1 -- 172.21.15.88:0/2845016713 <== mon.2 v1:172.21.15.88:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f56b0001cb0 con 0x7f56c00aad20 2024-03-29T10:43:47.101 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:47.115+0000 7f56c4c0e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:43:47.101 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:47.115+0000 7f56c4c0e640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:43:47.101 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:47.115+0000 7f56c4c0e640 10 monclient: dump: 2024-03-29T10:43:47.101 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:43:47.101 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:43:47.101 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:43:47.101 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:43:47.101 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:43:47.101 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:43:47.101 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:43:47.101 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:43:47.101 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:43:47.102 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:43:47.102 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:47.115+0000 7f56c6c12640 10 monclient: get_monmap_and_config success 2024-03-29T10:43:47.102 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:47.115+0000 7f56c6c12640 4 set_mon_vals no callback set 2024-03-29T10:43:47.102 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:47.116+0000 7f56c6c12640 10 monclient: shutdown 2024-03-29T10:43:47.102 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:47.116+0000 7f56c6c12640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:43:47.102 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:47.116+0000 7f56c6c12640 1 -- 172.21.15.88:0/2845016713 >> v1:172.21.15.88:6790/0 conn(0x7f56c00aad20 legacy=0x7f56c0151bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:47.102 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:47.116+0000 7f56c6c12640 1 -- 172.21.15.88:0/2845016713 shutdown_connections 2024-03-29T10:43:47.102 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:47.116+0000 7f56c6c12640 1 -- 172.21.15.88:0/2845016713 >> 172.21.15.88:0/2845016713 conn(0x7f56c00673e0 msgr2=0x7f56c00b3820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:43:47.102 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:47.116+0000 7f56c6c12640 1 -- 172.21.15.88:0/2845016713 shutdown_connections 2024-03-29T10:43:47.104 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:47.119+0000 7f56c6c12640 1 -- 172.21.15.88:0/2845016713 wait complete. 2024-03-29T10:43:47.104 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:47.120+0000 7f56c6c12640 10 monclient: build_initial_monmap 2024-03-29T10:43:47.104 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:47.120+0000 7f56c6c12640 10 monclient: monmap: 2024-03-29T10:43:47.104 INFO:teuthology.misc.health.smithi088.stderr:epoch 0 2024-03-29T10:43:47.104 INFO:teuthology.misc.health.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:43:47.104 INFO:teuthology.misc.health.smithi088.stderr:last_changed 0.000000 2024-03-29T10:43:47.104 INFO:teuthology.misc.health.smithi088.stderr:created 0.000000 2024-03-29T10:43:47.104 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 0 (unknown) 2024-03-29T10:43:47.104 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:43:47.105 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-03-29T10:43:47.105 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-29T10:43:47.105 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-c 2024-03-29T10:43:47.105 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:43:47.105 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:47.120+0000 7f56c6c12640 1 librados: starting msgr at 2024-03-29T10:43:47.105 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:47.120+0000 7f56c6c12640 1 librados: starting objecter 2024-03-29T10:43:47.105 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:47.121+0000 7f56c6c12640 1 Processor -- start 2024-03-29T10:43:47.105 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:47.121+0000 7f56c6c12640 1 -- start start 2024-03-29T10:43:47.105 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:47.121+0000 7f56c6c12640 1 librados: setting wanted keys 2024-03-29T10:43:47.105 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:47.121+0000 7f56c6c12640 1 librados: calling monclient init 2024-03-29T10:43:47.105 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:47.121+0000 7f56c6c12640 10 monclient: init 2024-03-29T10:43:47.106 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:47.122+0000 7f56c6c12640 10 monclient: _reopen_session rank -1 2024-03-29T10:43:47.106 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:47.122+0000 7f56c6c12640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:43:47.106 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:47.122+0000 7f56c6c12640 10 monclient(hunting): picked mon.noname-c con 0x7f56c00a9750 addr v1:172.21.15.88:6790/0 2024-03-29T10:43:47.109 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:47.125+0000 7f56c6c12640 10 monclient(hunting): picked mon.noname-a con 0x7f56c00aad20 addr v1:172.21.15.88:6789/0 2024-03-29T10:43:47.109 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:47.125+0000 7f56c540f640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f56c00aad20 0x7f56c01de980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:56083/0 (socket says 172.21.15.88:56083) 2024-03-29T10:43:47.109 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:47.125+0000 7f56c540f640 1 -- 172.21.15.88:0/3217241612 learned_addr learned my addr 172.21.15.88:0/3217241612 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-29T10:43:47.109 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:47.126+0000 7f56b67fc640 10 client.?.objecter ms_handle_connect 0x7f56c00aad20 2024-03-29T10:43:47.109 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:47.126+0000 7f56b67fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:43:47.115 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:47.126+0000 7f56c6c12640 10 monclient(hunting): picked mon.noname-b con 0x7f56c01df090 addr v1:172.21.15.196:6789/0 2024-03-29T10:43:47.115 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:47.127+0000 7f56b67fc640 10 client.?.objecter ms_handle_connect 0x7f56c00a9750 2024-03-29T10:43:47.115 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:47.127+0000 7f56b67fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:43:47.115 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:47.127+0000 7f56c6c12640 1 -- 172.21.15.88:0/3217241612 --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56c013ef50 con 0x7f56c00aad20 2024-03-29T10:43:47.116 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:47.127+0000 7f56c6c12640 1 -- 172.21.15.88:0/3217241612 --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56c01df520 con 0x7f56c01df090 2024-03-29T10:43:47.116 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:47.127+0000 7f56c6c12640 1 -- 172.21.15.88:0/3217241612 --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56c006f0c0 con 0x7f56c00a9750 2024-03-29T10:43:47.116 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:47.127+0000 7f56c6c12640 10 monclient(hunting): _renew_subs 2024-03-29T10:43:47.116 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:47.127+0000 7f56c6c12640 10 monclient(hunting): authenticate will time out at 949.363827s 2024-03-29T10:43:47.116 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:47.128+0000 7f56b67fc640 1 -- 172.21.15.88:0/3217241612 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1789735638 0 0) 0x7f56c006f0c0 con 0x7f56c00a9750 2024-03-29T10:43:47.116 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:47.128+0000 7f56b67fc640 10 monclient(hunting): _negotiate 2024-03-29T10:43:47.116 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:47.128+0000 7f56b67fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:43:47.116 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:47.128+0000 7f56b67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:43:47.116 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:47.128+0000 7f56b67fc640 10 monclient(hunting): my global_id is 4358 2024-03-29T10:43:47.116 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:47.128+0000 7f56b67fc640 1 -- 172.21.15.88:0/3217241612 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f56a4004eb0 con 0x7f56c00a9750 2024-03-29T10:43:47.116 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:47.128+0000 7f56b67fc640 1 -- 172.21.15.88:0/3217241612 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1587371863 0 0) 0x7f56bc001100 con 0x7f56c00aad20 2024-03-29T10:43:47.116 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:47.128+0000 7f56b67fc640 10 monclient(hunting): _negotiate 2024-03-29T10:43:47.116 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:47.128+0000 7f56b67fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:43:47.116 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:47.128+0000 7f56b67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:43:47.117 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:47.128+0000 7f56b67fc640 10 monclient(hunting): my global_id is 4326 2024-03-29T10:43:47.117 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:47.128+0000 7f56b67fc640 1 -- 172.21.15.88:0/3217241612 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f56c006f0c0 con 0x7f56c00aad20 2024-03-29T10:43:47.117 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:47.129+0000 7f56b67fc640 1 -- 172.21.15.88:0/3217241612 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3070273049 0 0) 0x7f56c006f0c0 con 0x7f56c00aad20 2024-03-29T10:43:47.117 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:47.129+0000 7f56b67fc640 1 -- 172.21.15.88:0/3217241612 --> v1:172.21.15.88:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f56bc001100 con 0x7f56c00aad20 2024-03-29T10:43:47.117 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:47.129+0000 7f56b67fc640 1 -- 172.21.15.88:0/3217241612 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f56bc001870 con 0x7f56c00aad20 2024-03-29T10:43:47.117 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:47.129+0000 7f56b67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:43:47.117 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:47.129+0000 7f56b67fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:43:47.117 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:47.129+0000 7f56b67fc640 10 monclient(hunting): dump: 2024-03-29T10:43:47.117 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:43:47.117 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:43:47.117 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:43:47.117 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:43:47.117 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:43:47.117 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:43:47.117 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:43:47.118 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:43:47.118 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:43:47.118 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:43:47.118 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:47.129+0000 7f56b67fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:43:47.118 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:47.129+0000 7f56b67fc640 1 -- 172.21.15.88:0/3217241612 <== mon.0 v1:172.21.15.88:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 967468969 0 0) 0x7f56bc001100 con 0x7f56c00aad20 2024-03-29T10:43:47.118 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:47.129+0000 7f56b67fc640 1 -- 172.21.15.88:0/3217241612 >> v1:172.21.15.88:6790/0 conn(0x7f56c00a9750 legacy=0x7f56c006e880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:47.118 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:47.129+0000 7f56b67fc640 1 -- 172.21.15.88:0/3217241612 >> v1:172.21.15.196:6789/0 conn(0x7f56c01df090 legacy=0x7f56c0071a30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:43:47.118 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:47.130+0000 7f56b67fc640 10 monclient: _finish_hunting 0 2024-03-29T10:43:47.118 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:47.130+0000 7f56b67fc640 1 monclient: found mon.a 2024-03-29T10:43:47.118 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:47.130+0000 7f56b67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:43:47.118 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:47.130+0000 7f56b67fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-03-29T10:43:47.118 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:47.130+0000 7f56b67fc640 1 -- 172.21.15.88:0/3217241612 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56c0070140 con 0x7f56c00aad20 2024-03-29T10:43:47.118 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:47.130+0000 7f56b67fc640 10 monclient: _finish_auth 0 2024-03-29T10:43:47.118 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:47.130+0000 7f56b67fc640 10 monclient: _check_auth_tickets 2024-03-29T10:43:47.119 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:47.130+0000 7f56b67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:43:47.119 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:47.130+0000 7f56b67fc640 1 -- 172.21.15.88:0/3217241612 <== mon.0 v1:172.21.15.88:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f56bc001d60 con 0x7f56c00aad20 2024-03-29T10:43:47.119 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:47.130+0000 7f56b67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:43:47.119 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:47.130+0000 7f56b67fc640 1 -- 172.21.15.88:0/3217241612 <== mon.0 v1:172.21.15.88:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f56bc0037b0 con 0x7f56c00aad20 2024-03-29T10:43:47.119 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:47.130+0000 7f56b67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:43:47.119 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:47.130+0000 7f56b67fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:43:47.119 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:47.130+0000 7f56b67fc640 10 monclient: dump: 2024-03-29T10:43:47.119 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:43:47.119 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:43:47.119 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:43:47.119 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:43:47.119 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:43:47.119 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:43:47.119 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:43:47.119 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:43:47.120 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:43:47.120 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:43:47.120 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:47.130+0000 7f56b7fff640 4 set_mon_vals no callback set 2024-03-29T10:43:47.120 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:47.131+0000 7f56c6c12640 5 monclient: authenticate success, global_id 4326 2024-03-29T10:43:47.120 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:47.131+0000 7f56c6c12640 10 monclient: _renew_subs 2024-03-29T10:43:47.120 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:47.131+0000 7f56c6c12640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-03-29T10:43:47.120 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:47.131+0000 7f56c6c12640 1 -- 172.21.15.88:0/3217241612 --> v1:172.21.15.88:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56c006e1c0 con 0x7f56c00aad20 2024-03-29T10:43:47.120 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:47.131+0000 7f56c6c12640 10 client.4326.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:43:47.120 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:47.131+0000 7f56c6c12640 10 monclient: _renew_subs 2024-03-29T10:43:47.120 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:47.131+0000 7f56c6c12640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-03-29T10:43:47.121 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:47.131+0000 7f56c6c12640 1 -- 172.21.15.88:0/3217241612 --> v1:172.21.15.88:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f56c0072320 con 0x7f56c00aad20 2024-03-29T10:43:47.121 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:47.131+0000 7f56c6c12640 1 librados: init done 2024-03-29T10:43:47.122 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:47.134+0000 7f56a3fff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:43:47.122 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:47.134+0000 7f56b67fc640 1 -- 172.21.15.88:0/3217241612 <== mon.0 v1:172.21.15.88:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4235629220 0 0) 0x7f56bc003910 con 0x7f56c00aad20 2024-03-29T10:43:47.122 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:47.134+0000 7f56a3fff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:43:47.122 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:47.134+0000 7f56a3fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-03-29T10:43:47.122 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:47.134+0000 7f56a3fff640 1 -- 172.21.15.88:0/3217241612 --> v1:172.21.15.88:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f569c005080 con 0x7f56c00aad20 2024-03-29T10:43:47.122 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:47.135+0000 7f56b67fc640 1 -- 172.21.15.88:0/3217241612 <== mon.0 v1:172.21.15.88:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4257673582 0 0) 0x7f56bc084910 con 0x7f56c00aad20 2024-03-29T10:43:47.122 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:47.135+0000 7f56b67fc640 10 client.4326.objecter ms_dispatch 0x7f56c0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:43:47.122 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:47.135+0000 7f56b67fc640 3 client.4326.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:43:47.122 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:47.135+0000 7f56b67fc640 3 client.4326.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:43:47.122 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:47.135+0000 7f56b67fc640 20 client.4326.objecter dump_active .. 0 homeless 2024-03-29T10:43:47.123 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:47.135+0000 7f56c5c10640 10 monclient: get_auth_request con 0x7f56a406db20 auth_method 0 2024-03-29T10:43:47.123 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:47.135+0000 7f56b67fc640 10 client.4326.objecter ms_handle_connect 0x7f56a406db20 2024-03-29T10:43:47.129 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:47.145+0000 7f56b67fc640 1 -- 172.21.15.88:0/3217241612 <== mon.0 v1:172.21.15.88:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f56bc050250 con 0x7f56c00aad20 2024-03-29T10:43:47.129 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:47.145+0000 7f56b67fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:43:47.129 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:47.145+0000 7f56b67fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:43:47.453 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:47.469+0000 7f56a3fff640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:43:47.453 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:47.469+0000 7f56a3fff640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:43:47.453 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:47.469+0000 7f56a3fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-03-29T10:43:47.453 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:47.469+0000 7f56a3fff640 1 -- 172.21.15.88:0/3217241612 --> v1:172.21.15.88:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f569c005330 con 0x7f56c00aad20 2024-03-29T10:43:47.453 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:47.470+0000 7f56b67fc640 1 -- 172.21.15.88:0/3217241612 <== mon.0 v1:172.21.15.88:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 344506374) 0x7f56bc052180 con 0x7f56c00aad20 2024-03-29T10:43:47.453 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:47.470+0000 7f56b67fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:43:47.454 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:47.470+0000 7f56b67fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:43:47.457 INFO:teuthology.misc.health.smithi088.stdout: 2024-03-29T10:43:47.457 INFO:teuthology.misc.health.smithi088.stdout:{"epoch":10,"fsid":"869f5188-f92b-42dd-a46d-8c646e0388a9","created":"2024-03-29T10:40:40.285087+0000","modified":"2024-03-29T10:40:42.536230+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:40:42.536230+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a26c0c1c-8de0-438c-a3f6-3df10ea61574","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5de24915-8dec-49b6-8b16-f3170197c591","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ac054af3-d589-4565-9644-e6af75975c53","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3ec4f045-c714-4a22-bd1a-e07330b45ab3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"cddb9867-8e30-4951-9b13-3d1e78b9c45a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dc6d9911-785d-432f-8b53-56a1ccab302f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"77c88437-b587-496e-8b4d-1824ea8b5305","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ce963d96-5a8e-44c3-99de-c0dbebe234ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:43:47.460 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:47.476+0000 7f56a3fff640 10 librados: watch_flush enter 2024-03-29T10:43:47.460 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:47.477+0000 7f56a3fff640 10 librados: watch_flush exit 2024-03-29T10:43:47.460 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:47.477+0000 7f56a3fff640 20 client.4326.objecter shutdown clearing up homeless session... 2024-03-29T10:43:47.460 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:47.477+0000 7f56a3fff640 10 client.4326.objecter successfully canceled tick 2024-03-29T10:43:47.460 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:47.477+0000 7f56a3fff640 1 -- 172.21.15.88:0/3217241612 >> v1:172.21.15.88:6816/1752861349 conn(0x7f56a406db20 legacy=0x7f56a406ffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:47.460 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:47.477+0000 7f56a3fff640 10 monclient: shutdown 2024-03-29T10:43:47.460 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:47.477+0000 7f56a3fff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:43:47.460 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:47.477+0000 7f56a3fff640 1 -- 172.21.15.88:0/3217241612 >> v1:172.21.15.88:6789/0 conn(0x7f56c00aad20 legacy=0x7f56c01de980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:47.460 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:47.477+0000 7f56a3fff640 1 -- 172.21.15.88:0/3217241612 shutdown_connections 2024-03-29T10:43:47.460 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:47.477+0000 7f56a3fff640 1 -- 172.21.15.88:0/3217241612 >> 172.21.15.88:0/3217241612 conn(0x7f56c00673e0 msgr2=0x7f56c0141880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:43:47.461 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:47.478+0000 7f56a3fff640 1 -- 172.21.15.88:0/3217241612 shutdown_connections 2024-03-29T10:43:47.461 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:47.478+0000 7f56a3fff640 1 -- 172.21.15.88:0/3217241612 wait complete. 2024-03-29T10:43:47.461 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:47.478+0000 7f56a3fff640 1 librados: shutdown 2024-03-29T10:43:47.478 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:43:54.285 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:43:54.433 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:54.450+0000 7f4490e7b640 10 monclient: get_monmap_and_config 2024-03-29T10:43:54.434 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:54.450+0000 7f4490e7b640 10 monclient: build_initial_monmap 2024-03-29T10:43:54.434 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:54.450+0000 7f4490e7b640 10 monclient: monmap: 2024-03-29T10:43:54.434 INFO:teuthology.misc.health.smithi088.stderr:epoch 0 2024-03-29T10:43:54.434 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:43:54.434 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:43:54.451698+0000 2024-03-29T10:43:54.434 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:43:54.451698+0000 2024-03-29T10:43:54.434 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 0 (unknown) 2024-03-29T10:43:54.434 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:43:54.434 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-03-29T10:43:54.434 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-29T10:43:54.434 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-b 2024-03-29T10:43:54.434 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:43:54.435 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:54.451+0000 7f4490e7b640 1 Processor -- start 2024-03-29T10:43:54.435 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:54.451+0000 7f4490e7b640 1 -- start start 2024-03-29T10:43:54.435 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:54.451+0000 7f4490e7b640 10 monclient: init 2024-03-29T10:43:54.438 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:54.452+0000 7f4490e7b640 10 monclient: _reopen_session rank -1 2024-03-29T10:43:54.438 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:54.452+0000 7f4490e7b640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:43:54.438 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:54.452+0000 7f4490e7b640 10 monclient(hunting): picked mon.noname-c con 0x7f448c0b0860 addr v1:172.21.15.196:6789/0 2024-03-29T10:43:54.438 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:54.452+0000 7f4490e7b640 10 monclient(hunting): picked mon.noname-b con 0x7f448c072450 addr v1:172.21.15.88:6790/0 2024-03-29T10:43:54.438 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:54.453+0000 7f4490e7b640 10 monclient(hunting): picked mon.noname-a con 0x7f448c13ee40 addr v1:172.21.15.88:6789/0 2024-03-29T10:43:54.438 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:54.453+0000 7f4490e7b640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f448c0ad840 con 0x7f448c13ee40 2024-03-29T10:43:54.438 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:54.453+0000 7f4490e7b640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f448c13c830 con 0x7f448c0b0860 2024-03-29T10:43:54.438 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:54.453+0000 7f4490e7b640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f448c13c9c0 con 0x7f448c072450 2024-03-29T10:43:54.438 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:54.453+0000 7f4490e7b640 10 monclient(hunting): _renew_subs 2024-03-29T10:43:54.439 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:54.453+0000 7f4490e7b640 10 monclient(hunting): authenticate will time out at 956.689182s 2024-03-29T10:43:54.439 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:54.453+0000 7f448ad9d640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f448c13ee40 0x7f448c13f260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:42706/0 (socket says 172.21.15.88:42706) 2024-03-29T10:43:54.439 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:54.453+0000 7f448ad9d640 1 -- 172.21.15.88:0/3107696896 learned_addr learned my addr 172.21.15.88:0/3107696896 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-29T10:43:54.439 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:54.454+0000 7f448959a640 1 -- 172.21.15.88:0/3107696896 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 938825862 0 0) 0x7f448c0ad840 con 0x7f448c13ee40 2024-03-29T10:43:54.439 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:54.454+0000 7f448959a640 10 monclient(hunting): _negotiate 2024-03-29T10:43:54.439 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:54.454+0000 7f448959a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:43:54.439 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:54.454+0000 7f448959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:43:54.439 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:54.454+0000 7f448959a640 10 monclient(hunting): my global_id is 4329 2024-03-29T10:43:54.439 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:54.454+0000 7f448959a640 1 -- 172.21.15.88:0/3107696896 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4474003520 con 0x7f448c13ee40 2024-03-29T10:43:54.440 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:54.454+0000 7f448959a640 1 -- 172.21.15.88:0/3107696896 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1462651910 0 0) 0x7f448c13c9c0 con 0x7f448c072450 2024-03-29T10:43:54.440 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:54.454+0000 7f448959a640 10 monclient(hunting): _negotiate 2024-03-29T10:43:54.440 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:54.454+0000 7f448959a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:43:54.440 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:54.454+0000 7f448959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:43:54.440 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:54.454+0000 7f448959a640 10 monclient(hunting): my global_id is 4361 2024-03-29T10:43:54.440 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:54.454+0000 7f448959a640 1 -- 172.21.15.88:0/3107696896 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f448c0ad840 con 0x7f448c072450 2024-03-29T10:43:54.440 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:54.454+0000 7f448959a640 1 -- 172.21.15.88:0/3107696896 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3643867648 0 0) 0x7f448c13c830 con 0x7f448c0b0860 2024-03-29T10:43:54.440 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:54.454+0000 7f448959a640 10 monclient(hunting): _negotiate 2024-03-29T10:43:54.440 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:54.454+0000 7f448959a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:43:54.440 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:54.454+0000 7f448959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:43:54.440 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:54.454+0000 7f448959a640 10 monclient(hunting): my global_id is 4324 2024-03-29T10:43:54.440 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:54.454+0000 7f448959a640 1 -- 172.21.15.88:0/3107696896 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f448c13c9c0 con 0x7f448c0b0860 2024-03-29T10:43:54.441 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:54.454+0000 7f448959a640 1 -- 172.21.15.88:0/3107696896 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4251157085 0 0) 0x7f448c0ad840 con 0x7f448c072450 2024-03-29T10:43:54.441 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:54.454+0000 7f448959a640 1 -- 172.21.15.88:0/3107696896 >> v1:172.21.15.196:6789/0 conn(0x7f448c0b0860 legacy=0x7f448c0acfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:54.441 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:54.454+0000 7f448959a640 1 -- 172.21.15.88:0/3107696896 >> v1:172.21.15.88:6789/0 conn(0x7f448c13ee40 legacy=0x7f448c13f260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:54.441 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:54.454+0000 7f448959a640 10 monclient: _finish_hunting 0 2024-03-29T10:43:54.441 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:54.454+0000 7f448959a640 1 monclient: found mon.noname-b 2024-03-29T10:43:54.441 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:54.454+0000 7f448959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:43:54.441 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:54.454+0000 7f448959a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.88:6790/0 2024-03-29T10:43:54.441 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:54.454+0000 7f448959a640 1 -- 172.21.15.88:0/3107696896 --> v1:172.21.15.88:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f448c13f970 con 0x7f448c072450 2024-03-29T10:43:54.441 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:54.454+0000 7f448959a640 10 monclient: _finish_auth 0 2024-03-29T10:43:54.441 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:54.454+0000 7f448959a640 10 monclient: _check_auth_tickets 2024-03-29T10:43:54.441 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:54.454+0000 7f448959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:43:54.442 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:54.454+0000 7f448959a640 1 -- 172.21.15.88:0/3107696896 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f447c001670 con 0x7f448c072450 2024-03-29T10:43:54.442 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:54.454+0000 7f4490e7b640 5 monclient: authenticate success, global_id 4361 2024-03-29T10:43:54.442 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:54.454+0000 7f448959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:43:54.442 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:54.454+0000 7f448959a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:43:54.442 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:54.454+0000 7f448959a640 10 monclient: dump: 2024-03-29T10:43:54.442 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:43:54.442 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:43:54.442 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:43:54.442 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:43:54.442 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:43:54.442 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:43:54.443 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:43:54.443 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:43:54.443 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:43:54.443 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:43:54.443 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:54.454+0000 7f4490e7b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:43:54.443 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:54.455+0000 7f448959a640 1 -- 172.21.15.88:0/3107696896 <== mon.2 v1:172.21.15.88:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f447c001850 con 0x7f448c072450 2024-03-29T10:43:54.443 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:54.455+0000 7f448959a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:43:54.443 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:54.455+0000 7f448959a640 1 -- 172.21.15.88:0/3107696896 <== mon.2 v1:172.21.15.88:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f447c001c60 con 0x7f448c072450 2024-03-29T10:43:54.443 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:54.455+0000 7f448959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:43:54.443 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:54.455+0000 7f448959a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:43:54.443 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:54.455+0000 7f448959a640 10 monclient: dump: 2024-03-29T10:43:54.443 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:43:54.444 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:43:54.444 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:43:54.444 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:43:54.444 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:43:54.444 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:43:54.444 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:43:54.444 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:43:54.444 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:43:54.444 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:43:54.444 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:54.455+0000 7f4490e7b640 10 monclient: get_monmap_and_config success 2024-03-29T10:43:54.444 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:54.455+0000 7f4490e7b640 4 set_mon_vals no callback set 2024-03-29T10:43:54.450 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:54.456+0000 7f4490e7b640 10 monclient: shutdown 2024-03-29T10:43:54.451 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:54.456+0000 7f4490e7b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:43:54.451 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:54.456+0000 7f4490e7b640 1 -- 172.21.15.88:0/3107696896 >> v1:172.21.15.88:6790/0 conn(0x7f448c072450 legacy=0x7f448c13c120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:54.451 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:54.456+0000 7f4490e7b640 1 -- 172.21.15.88:0/3107696896 shutdown_connections 2024-03-29T10:43:54.451 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:54.456+0000 7f4490e7b640 1 -- 172.21.15.88:0/3107696896 >> 172.21.15.88:0/3107696896 conn(0x7f448c06b660 msgr2=0x7f448c06ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:43:54.451 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:54.456+0000 7f4490e7b640 1 -- 172.21.15.88:0/3107696896 shutdown_connections 2024-03-29T10:43:54.451 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:54.456+0000 7f4490e7b640 1 -- 172.21.15.88:0/3107696896 wait complete. 2024-03-29T10:43:54.451 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:54.457+0000 7f4490e7b640 10 monclient: build_initial_monmap 2024-03-29T10:43:54.451 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:54.457+0000 7f4490e7b640 10 monclient: monmap: 2024-03-29T10:43:54.451 INFO:teuthology.misc.health.smithi088.stderr:epoch 0 2024-03-29T10:43:54.451 INFO:teuthology.misc.health.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:43:54.451 INFO:teuthology.misc.health.smithi088.stderr:last_changed 0.000000 2024-03-29T10:43:54.451 INFO:teuthology.misc.health.smithi088.stderr:created 0.000000 2024-03-29T10:43:54.451 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 0 (unknown) 2024-03-29T10:43:54.452 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:43:54.452 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-03-29T10:43:54.452 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-29T10:43:54.452 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-c 2024-03-29T10:43:54.452 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:43:54.452 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:54.458+0000 7f4490e7b640 1 librados: starting msgr at 2024-03-29T10:43:54.452 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:54.458+0000 7f4490e7b640 1 librados: starting objecter 2024-03-29T10:43:54.452 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:54.458+0000 7f4490e7b640 1 Processor -- start 2024-03-29T10:43:54.452 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:54.458+0000 7f4490e7b640 1 -- start start 2024-03-29T10:43:54.453 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:54.458+0000 7f4490e7b640 1 librados: setting wanted keys 2024-03-29T10:43:54.453 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:54.458+0000 7f4490e7b640 1 librados: calling monclient init 2024-03-29T10:43:54.453 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:54.458+0000 7f4490e7b640 10 monclient: init 2024-03-29T10:43:54.453 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:54.458+0000 7f4490e7b640 10 monclient: _reopen_session rank -1 2024-03-29T10:43:54.453 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:54.458+0000 7f4490e7b640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:43:54.453 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:54.459+0000 7f4490e7b640 10 monclient(hunting): picked mon.noname-b con 0x7f448c072450 addr v1:172.21.15.196:6789/0 2024-03-29T10:43:54.453 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:54.459+0000 7f4490e7b640 10 monclient(hunting): picked mon.noname-a con 0x7f448c0b0860 addr v1:172.21.15.88:6789/0 2024-03-29T10:43:54.453 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:54.459+0000 7f4490e7b640 10 monclient(hunting): picked mon.noname-c con 0x7f448c13ee40 addr v1:172.21.15.88:6790/0 2024-03-29T10:43:54.453 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:54.459+0000 7f4490e7b640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f448c1cb3e0 con 0x7f448c0b0860 2024-03-29T10:43:54.454 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:54.459+0000 7f4490e7b640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f448c13cbb0 con 0x7f448c072450 2024-03-29T10:43:54.454 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:54.459+0000 7f4490e7b640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f448c13cd40 con 0x7f448c13ee40 2024-03-29T10:43:54.454 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:54.459+0000 7f4490e7b640 10 monclient(hunting): _renew_subs 2024-03-29T10:43:54.454 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:54.459+0000 7f4489d9b640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f448c0b0860 0x7f448c1d7780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:42922/0 (socket says 172.21.15.88:42922) 2024-03-29T10:43:54.454 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:54.459+0000 7f4489d9b640 1 -- 172.21.15.88:0/1154384220 learned_addr learned my addr 172.21.15.88:0/1154384220 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-29T10:43:54.454 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:54.459+0000 7f447affd640 10 client.?.objecter ms_handle_connect 0x7f448c0b0860 2024-03-29T10:43:54.454 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:54.459+0000 7f447affd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:43:54.454 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:54.459+0000 7f447affd640 10 client.?.objecter ms_handle_connect 0x7f448c072450 2024-03-29T10:43:54.454 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:54.459+0000 7f447affd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:43:54.454 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:54.459+0000 7f4490e7b640 10 monclient(hunting): authenticate will time out at 956.695155s 2024-03-29T10:43:54.454 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:54.459+0000 7f447affd640 1 -- 172.21.15.88:0/1154384220 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3981899047 0 0) 0x7f448c1cb3e0 con 0x7f448c0b0860 2024-03-29T10:43:54.454 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:54.459+0000 7f447affd640 10 monclient(hunting): _negotiate 2024-03-29T10:43:54.454 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:54.459+0000 7f447affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:43:54.455 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:54.459+0000 7f447affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:43:54.455 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:54.459+0000 7f447affd640 10 monclient(hunting): my global_id is 4332 2024-03-29T10:43:54.455 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:54.459+0000 7f447affd640 1 -- 172.21.15.88:0/1154384220 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f446c004eb0 con 0x7f448c0b0860 2024-03-29T10:43:54.455 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:54.459+0000 7f447affd640 10 client.?.objecter ms_handle_connect 0x7f448c13ee40 2024-03-29T10:43:54.455 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:54.459+0000 7f447affd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:43:54.455 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:54.459+0000 7f447affd640 1 -- 172.21.15.88:0/1154384220 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3748767039 0 0) 0x7f448c13cbb0 con 0x7f448c072450 2024-03-29T10:43:54.455 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:54.459+0000 7f447affd640 10 monclient(hunting): _negotiate 2024-03-29T10:43:54.455 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:54.459+0000 7f447affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:43:54.455 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:54.459+0000 7f447affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:43:54.455 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:54.459+0000 7f447affd640 10 monclient(hunting): my global_id is 4327 2024-03-29T10:43:54.455 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:54.459+0000 7f447affd640 1 -- 172.21.15.88:0/1154384220 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f448c1cb3e0 con 0x7f448c072450 2024-03-29T10:43:54.455 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:54.460+0000 7f447affd640 1 -- 172.21.15.88:0/1154384220 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3826562814 0 0) 0x7f446c004eb0 con 0x7f448c0b0860 2024-03-29T10:43:54.455 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:54.460+0000 7f447affd640 1 -- 172.21.15.88:0/1154384220 --> v1:172.21.15.88:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f448c13cbb0 con 0x7f448c0b0860 2024-03-29T10:43:54.455 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:54.460+0000 7f447affd640 1 -- 172.21.15.88:0/1154384220 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f447c001c60 con 0x7f448c0b0860 2024-03-29T10:43:54.456 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:54.460+0000 7f447affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:43:54.456 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:54.460+0000 7f447affd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:43:54.456 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:54.460+0000 7f447affd640 10 monclient(hunting): dump: 2024-03-29T10:43:54.456 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:43:54.456 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:43:54.456 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:43:54.456 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:43:54.456 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:43:54.456 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:43:54.456 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:43:54.456 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:43:54.456 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:43:54.456 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:43:54.456 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:54.460+0000 7f447affd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:43:54.456 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:54.460+0000 7f447affd640 1 -- 172.21.15.88:0/1154384220 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2368968270 0 0) 0x7f4474003520 con 0x7f448c13ee40 2024-03-29T10:43:54.457 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:54.460+0000 7f447affd640 10 monclient(hunting): _negotiate 2024-03-29T10:43:54.457 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:54.460+0000 7f447affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:43:54.457 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:54.460+0000 7f447affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:43:54.457 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:54.460+0000 7f447affd640 10 monclient(hunting): my global_id is 4364 2024-03-29T10:43:54.457 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:54.460+0000 7f447affd640 1 -- 172.21.15.88:0/1154384220 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f446c004eb0 con 0x7f448c13ee40 2024-03-29T10:43:54.457 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:54.460+0000 7f447affd640 1 -- 172.21.15.88:0/1154384220 <== mon.0 v1:172.21.15.88:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1851874328 0 0) 0x7f448c13cbb0 con 0x7f448c0b0860 2024-03-29T10:43:54.457 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:54.460+0000 7f447affd640 1 -- 172.21.15.88:0/1154384220 >> v1:172.21.15.88:6790/0 conn(0x7f448c13ee40 legacy=0x7f448c1e1cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:54.457 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:54.460+0000 7f447affd640 1 -- 172.21.15.88:0/1154384220 >> v1:172.21.15.196:6789/0 conn(0x7f448c072450 legacy=0x7f448c1c94b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:54.457 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:54.460+0000 7f447affd640 10 monclient: _finish_hunting 0 2024-03-29T10:43:54.457 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:54.460+0000 7f447affd640 1 monclient: found mon.a 2024-03-29T10:43:54.457 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:54.460+0000 7f447affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:43:54.457 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:54.460+0000 7f447affd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-03-29T10:43:54.457 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:54.460+0000 7f447affd640 1 -- 172.21.15.88:0/1154384220 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f448c1c9bc0 con 0x7f448c0b0860 2024-03-29T10:43:54.457 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:54.460+0000 7f447affd640 10 monclient: _finish_auth 0 2024-03-29T10:43:54.457 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:54.460+0000 7f447affd640 10 monclient: _check_auth_tickets 2024-03-29T10:43:54.458 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:54.460+0000 7f447affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:43:54.458 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:54.460+0000 7f4490e7b640 5 monclient: authenticate success, global_id 4332 2024-03-29T10:43:54.458 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:54.460+0000 7f4490e7b640 10 monclient: _renew_subs 2024-03-29T10:43:54.458 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:54.460+0000 7f4490e7b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-03-29T10:43:54.458 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:54.460+0000 7f4490e7b640 1 -- 172.21.15.88:0/1154384220 --> v1:172.21.15.88:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f448c1d7e90 con 0x7f448c0b0860 2024-03-29T10:43:54.458 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:54.461+0000 7f4490e7b640 10 client.4332.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:43:54.458 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:54.461+0000 7f4490e7b640 10 monclient: _renew_subs 2024-03-29T10:43:54.458 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:54.461+0000 7f4490e7b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-03-29T10:43:54.458 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:54.461+0000 7f4490e7b640 1 -- 172.21.15.88:0/1154384220 --> v1:172.21.15.88:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f448c1c8980 con 0x7f448c0b0860 2024-03-29T10:43:54.458 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:54.461+0000 7f4490e7b640 1 librados: init done 2024-03-29T10:43:54.458 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:54.461+0000 7f447affd640 1 -- 172.21.15.88:0/1154384220 <== mon.0 v1:172.21.15.88:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f447c003960 con 0x7f448c0b0860 2024-03-29T10:43:54.458 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:54.461+0000 7f447affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:43:54.458 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:54.461+0000 7f447affd640 1 -- 172.21.15.88:0/1154384220 <== mon.0 v1:172.21.15.88:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f447c001230 con 0x7f448c0b0860 2024-03-29T10:43:54.458 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:54.461+0000 7f447affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:43:54.458 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:54.461+0000 7f447affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:43:54.459 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:54.461+0000 7f447affd640 10 monclient: dump: 2024-03-29T10:43:54.459 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:43:54.459 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:43:54.459 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:43:54.459 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:43:54.459 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:43:54.459 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:43:54.459 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:43:54.459 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:43:54.459 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:43:54.459 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:43:54.459 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:54.461+0000 7f4488d99640 4 set_mon_vals no callback set 2024-03-29T10:43:54.459 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:54.462+0000 7f4490e7b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:43:54.459 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:54.462+0000 7f4490e7b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:43:54.459 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:54.462+0000 7f4490e7b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-03-29T10:43:54.459 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:54.462+0000 7f4490e7b640 1 -- 172.21.15.88:0/1154384220 --> v1:172.21.15.88:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4470005080 con 0x7f448c0b0860 2024-03-29T10:43:54.460 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:54.464+0000 7f447affd640 1 -- 172.21.15.88:0/1154384220 <== mon.0 v1:172.21.15.88:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4235629220 0 0) 0x7f447c003c70 con 0x7f448c0b0860 2024-03-29T10:43:54.460 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:54.465+0000 7f447affd640 1 -- 172.21.15.88:0/1154384220 <== mon.0 v1:172.21.15.88:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4257673582 0 0) 0x7f447c084be0 con 0x7f448c0b0860 2024-03-29T10:43:54.460 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:54.465+0000 7f447affd640 10 client.4332.objecter ms_dispatch 0x7f448c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:43:54.460 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:54.465+0000 7f447affd640 3 client.4332.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:43:54.460 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:54.465+0000 7f447affd640 3 client.4332.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:43:54.460 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:54.465+0000 7f447affd640 20 client.4332.objecter dump_active .. 0 homeless 2024-03-29T10:43:54.460 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:54.465+0000 7f448a59c640 10 monclient: get_auth_request con 0x7f446c06de50 auth_method 0 2024-03-29T10:43:54.460 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:54.465+0000 7f447affd640 10 client.4332.objecter ms_handle_connect 0x7f446c06de50 2024-03-29T10:43:54.460 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:54.472+0000 7f447affd640 1 -- 172.21.15.88:0/1154384220 <== mon.0 v1:172.21.15.88:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f447c0502f0 con 0x7f448c0b0860 2024-03-29T10:43:54.461 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:54.472+0000 7f447affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:43:54.461 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:54.472+0000 7f447affd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:43:54.806 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:54.822+0000 7f4490e7b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:43:54.806 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:54.822+0000 7f4490e7b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:43:54.806 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:54.822+0000 7f4490e7b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-03-29T10:43:54.806 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:54.822+0000 7f4490e7b640 1 -- 172.21.15.88:0/1154384220 --> v1:172.21.15.88:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4470005330 con 0x7f448c0b0860 2024-03-29T10:43:54.806 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:54.823+0000 7f447affd640 1 -- 172.21.15.88:0/1154384220 <== mon.0 v1:172.21.15.88:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 344506374) 0x7f447c0523f0 con 0x7f448c0b0860 2024-03-29T10:43:54.806 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:54.823+0000 7f447affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:43:54.807 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:54.823+0000 7f447affd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:43:54.807 INFO:teuthology.misc.health.smithi088.stdout: 2024-03-29T10:43:54.807 INFO:teuthology.misc.health.smithi088.stdout:{"epoch":10,"fsid":"869f5188-f92b-42dd-a46d-8c646e0388a9","created":"2024-03-29T10:40:40.285087+0000","modified":"2024-03-29T10:40:42.536230+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:40:42.536230+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a26c0c1c-8de0-438c-a3f6-3df10ea61574","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5de24915-8dec-49b6-8b16-f3170197c591","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ac054af3-d589-4565-9644-e6af75975c53","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3ec4f045-c714-4a22-bd1a-e07330b45ab3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"cddb9867-8e30-4951-9b13-3d1e78b9c45a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dc6d9911-785d-432f-8b53-56a1ccab302f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"77c88437-b587-496e-8b4d-1824ea8b5305","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ce963d96-5a8e-44c3-99de-c0dbebe234ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:43:54.810 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:54.827+0000 7f4490e7b640 10 librados: watch_flush enter 2024-03-29T10:43:54.811 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:54.827+0000 7f4490e7b640 10 librados: watch_flush exit 2024-03-29T10:43:54.811 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:54.828+0000 7f4490e7b640 20 client.4332.objecter shutdown clearing up homeless session... 2024-03-29T10:43:54.811 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:54.828+0000 7f4490e7b640 10 client.4332.objecter successfully canceled tick 2024-03-29T10:43:54.811 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:54.828+0000 7f4490e7b640 1 -- 172.21.15.88:0/1154384220 >> v1:172.21.15.88:6816/1752861349 conn(0x7f446c06de50 legacy=0x7f446c070310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:54.811 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:54.828+0000 7f4490e7b640 10 monclient: shutdown 2024-03-29T10:43:54.811 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:54.828+0000 7f4490e7b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:43:54.811 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:54.828+0000 7f4490e7b640 1 -- 172.21.15.88:0/1154384220 >> v1:172.21.15.88:6789/0 conn(0x7f448c0b0860 legacy=0x7f448c1d7780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:54.811 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:54.828+0000 7f4490e7b640 1 -- 172.21.15.88:0/1154384220 shutdown_connections 2024-03-29T10:43:54.811 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:54.828+0000 7f4490e7b640 1 -- 172.21.15.88:0/1154384220 >> 172.21.15.88:0/1154384220 conn(0x7f448c06b660 msgr2=0x7f448c0a9d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:43:54.812 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:54.828+0000 7f4490e7b640 1 -- 172.21.15.88:0/1154384220 shutdown_connections 2024-03-29T10:43:54.812 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:54.828+0000 7f4490e7b640 1 -- 172.21.15.88:0/1154384220 wait complete. 2024-03-29T10:43:54.812 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:43:54.829+0000 7f4490e7b640 1 librados: shutdown 2024-03-29T10:43:54.827 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:44:01.633 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:44:01.779 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:01.796+0000 7f3180225640 10 monclient: get_monmap_and_config 2024-03-29T10:44:01.779 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:01.796+0000 7f3180225640 10 monclient: build_initial_monmap 2024-03-29T10:44:01.779 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:01.796+0000 7f3180225640 10 monclient: monmap: 2024-03-29T10:44:01.780 INFO:teuthology.misc.health.smithi088.stderr:epoch 0 2024-03-29T10:44:01.780 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:44:01.780 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:44:01.797362+0000 2024-03-29T10:44:01.780 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:44:01.797362+0000 2024-03-29T10:44:01.780 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 0 (unknown) 2024-03-29T10:44:01.780 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:44:01.780 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-03-29T10:44:01.780 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-29T10:44:01.780 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-b 2024-03-29T10:44:01.780 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:44:01.781 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:01.797+0000 7f3180225640 1 Processor -- start 2024-03-29T10:44:01.781 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:01.797+0000 7f3180225640 1 -- start start 2024-03-29T10:44:01.781 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:01.797+0000 7f3180225640 10 monclient: init 2024-03-29T10:44:01.781 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:01.798+0000 7f3180225640 10 monclient: _reopen_session rank -1 2024-03-29T10:44:01.782 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:01.798+0000 7f3180225640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:44:01.782 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:01.798+0000 7f3180225640 10 monclient(hunting): picked mon.noname-b con 0x7f31780b0860 addr v1:172.21.15.88:6790/0 2024-03-29T10:44:01.782 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:01.798+0000 7f3180225640 10 monclient(hunting): picked mon.noname-c con 0x7f317813ef30 addr v1:172.21.15.196:6789/0 2024-03-29T10:44:01.782 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:01.798+0000 7f3180225640 10 monclient(hunting): picked mon.noname-a con 0x7f31780af8e0 addr v1:172.21.15.88:6789/0 2024-03-29T10:44:01.782 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:01.798+0000 7f3180225640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31780b0c40 con 0x7f31780af8e0 2024-03-29T10:44:01.782 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:01.798+0000 7f3180225640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31780aed10 con 0x7f317813ef30 2024-03-29T10:44:01.782 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:01.798+0000 7f3180225640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31780afd50 con 0x7f31780b0860 2024-03-29T10:44:01.782 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:01.798+0000 7f3180225640 10 monclient(hunting): _renew_subs 2024-03-29T10:44:01.782 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:01.798+0000 7f3180225640 10 monclient(hunting): authenticate will time out at 964.034294s 2024-03-29T10:44:01.784 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:01.801+0000 7f317e7c2640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f31780af8e0 0x7f3178149a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:35713/0 (socket says 172.21.15.88:35713) 2024-03-29T10:44:01.784 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:01.801+0000 7f317e7c2640 1 -- 172.21.15.88:0/3890535047 learned_addr learned my addr 172.21.15.88:0/3890535047 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-29T10:44:01.785 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:01.802+0000 7f317cfbf640 1 -- 172.21.15.88:0/3890535047 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3501021325 0 0) 0x7f31780b0c40 con 0x7f31780af8e0 2024-03-29T10:44:01.785 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:01.802+0000 7f317cfbf640 10 monclient(hunting): _negotiate 2024-03-29T10:44:01.785 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:01.802+0000 7f317cfbf640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:01.785 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:01.802+0000 7f317cfbf640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:01.785 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:01.802+0000 7f317cfbf640 10 monclient(hunting): my global_id is 4335 2024-03-29T10:44:01.785 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:01.802+0000 7f317cfbf640 1 -- 172.21.15.88:0/3890535047 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3164005730 con 0x7f31780af8e0 2024-03-29T10:44:01.785 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:01.802+0000 7f317cfbf640 1 -- 172.21.15.88:0/3890535047 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1625260148 0 0) 0x7f3164005730 con 0x7f31780af8e0 2024-03-29T10:44:01.785 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:01.802+0000 7f317cfbf640 1 -- 172.21.15.88:0/3890535047 >> v1:172.21.15.88:6790/0 conn(0x7f31780b0860 legacy=0x7f317813e820 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:44:01.787 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:01.804+0000 7f317cfbf640 1 -- 172.21.15.88:0/3890535047 >> v1:172.21.15.196:6789/0 conn(0x7f317813ef30 legacy=0x7f31780af1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:44:01.787 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:01.804+0000 7f317cfbf640 10 monclient: _finish_hunting 0 2024-03-29T10:44:01.788 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:01.804+0000 7f317cfbf640 1 monclient: found mon.noname-a 2024-03-29T10:44:01.789 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:01.804+0000 7f317cfbf640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:44:01.789 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:01.804+0000 7f317cfbf640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.88:6789/0 2024-03-29T10:44:01.789 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:01.804+0000 7f317cfbf640 1 -- 172.21.15.88:0/3890535047 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31780ab1b0 con 0x7f31780af8e0 2024-03-29T10:44:01.789 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:01.804+0000 7f317cfbf640 10 monclient: _finish_auth 0 2024-03-29T10:44:01.790 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:01.804+0000 7f317cfbf640 10 monclient: _check_auth_tickets 2024-03-29T10:44:01.790 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:01.804+0000 7f317cfbf640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:44:01.790 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:01.804+0000 7f3180225640 5 monclient: authenticate success, global_id 4335 2024-03-29T10:44:01.790 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:01.804+0000 7f3180225640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:44:01.790 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:01.804+0000 7f317cfbf640 1 -- 172.21.15.88:0/3890535047 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f3168001690 con 0x7f31780af8e0 2024-03-29T10:44:01.790 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:01.804+0000 7f317cfbf640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:01.790 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:01.804+0000 7f317cfbf640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:44:01.790 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:01.804+0000 7f317cfbf640 10 monclient: dump: 2024-03-29T10:44:01.790 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:44:01.790 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:44:01.790 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:44:01.790 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:44:01.790 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:01.791 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:44:01.791 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:44:01.791 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:44:01.791 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:44:01.791 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:44:01.791 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:01.804+0000 7f3180225640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:44:01.791 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:01.805+0000 7f317cfbf640 1 -- 172.21.15.88:0/3890535047 <== mon.0 v1:172.21.15.88:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3168001860 con 0x7f31780af8e0 2024-03-29T10:44:01.791 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:01.805+0000 7f317cfbf640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:44:01.791 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:01.805+0000 7f3180225640 10 monclient: get_monmap_and_config success 2024-03-29T10:44:01.791 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:01.805+0000 7f3180225640 4 set_mon_vals no callback set 2024-03-29T10:44:01.792 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:01.806+0000 7f317cfbf640 1 -- 172.21.15.88:0/3890535047 <== mon.0 v1:172.21.15.88:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f3168001cb0 con 0x7f31780af8e0 2024-03-29T10:44:01.792 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:01.806+0000 7f317cfbf640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:01.792 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:01.806+0000 7f317cfbf640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:44:01.792 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:01.806+0000 7f317cfbf640 10 monclient: dump: 2024-03-29T10:44:01.792 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:44:01.792 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:44:01.792 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:44:01.792 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:44:01.792 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:01.792 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:44:01.792 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:44:01.793 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:44:01.793 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:44:01.793 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:44:01.793 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:01.806+0000 7f3180225640 10 monclient: shutdown 2024-03-29T10:44:01.793 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:01.806+0000 7f3180225640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:44:01.793 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:01.806+0000 7f3180225640 1 -- 172.21.15.88:0/3890535047 >> v1:172.21.15.88:6789/0 conn(0x7f31780af8e0 legacy=0x7f3178149a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:01.793 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:01.806+0000 7f3180225640 1 -- 172.21.15.88:0/3890535047 shutdown_connections 2024-03-29T10:44:01.793 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:01.806+0000 7f3180225640 1 -- 172.21.15.88:0/3890535047 >> 172.21.15.88:0/3890535047 conn(0x7f317806b660 msgr2=0x7f317806ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:44:01.794 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:01.809+0000 7f3180225640 1 -- 172.21.15.88:0/3890535047 shutdown_connections 2024-03-29T10:44:01.795 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:01.809+0000 7f3180225640 1 -- 172.21.15.88:0/3890535047 wait complete. 2024-03-29T10:44:01.795 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:01.810+0000 7f3180225640 10 monclient: build_initial_monmap 2024-03-29T10:44:01.795 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:01.810+0000 7f3180225640 10 monclient: monmap: 2024-03-29T10:44:01.795 INFO:teuthology.misc.health.smithi088.stderr:epoch 0 2024-03-29T10:44:01.795 INFO:teuthology.misc.health.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:44:01.795 INFO:teuthology.misc.health.smithi088.stderr:last_changed 0.000000 2024-03-29T10:44:01.795 INFO:teuthology.misc.health.smithi088.stderr:created 0.000000 2024-03-29T10:44:01.795 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 0 (unknown) 2024-03-29T10:44:01.796 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:44:01.796 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-03-29T10:44:01.796 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-29T10:44:01.796 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-c 2024-03-29T10:44:01.796 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:44:01.796 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:01.811+0000 7f3180225640 1 librados: starting msgr at 2024-03-29T10:44:01.796 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:01.811+0000 7f3180225640 1 librados: starting objecter 2024-03-29T10:44:01.796 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:01.811+0000 7f3180225640 1 Processor -- start 2024-03-29T10:44:01.798 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:01.811+0000 7f3180225640 1 -- start start 2024-03-29T10:44:01.798 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:01.811+0000 7f3180225640 1 librados: setting wanted keys 2024-03-29T10:44:01.799 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:01.811+0000 7f3180225640 1 librados: calling monclient init 2024-03-29T10:44:01.799 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:01.811+0000 7f3180225640 10 monclient: init 2024-03-29T10:44:01.799 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:01.812+0000 7f3180225640 10 monclient: _reopen_session rank -1 2024-03-29T10:44:01.799 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:01.812+0000 7f3180225640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:44:01.799 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:01.812+0000 7f3180225640 10 monclient(hunting): picked mon.noname-b con 0x7f31780af8e0 addr v1:172.21.15.196:6789/0 2024-03-29T10:44:01.799 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:01.812+0000 7f3180225640 10 monclient(hunting): picked mon.noname-c con 0x7f31780b0860 addr v1:172.21.15.88:6790/0 2024-03-29T10:44:01.799 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:01.812+0000 7f3180225640 10 monclient(hunting): picked mon.noname-a con 0x7f317813ef30 addr v1:172.21.15.88:6789/0 2024-03-29T10:44:01.799 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:01.812+0000 7f3180225640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3178136d70 con 0x7f317813ef30 2024-03-29T10:44:01.799 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:01.812+0000 7f3180225640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3178138490 con 0x7f31780af8e0 2024-03-29T10:44:01.799 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:01.812+0000 7f3180225640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3178137300 con 0x7f31780b0860 2024-03-29T10:44:01.799 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:01.812+0000 7f3180225640 10 monclient(hunting): _renew_subs 2024-03-29T10:44:01.799 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:01.812+0000 7f3180225640 10 monclient(hunting): authenticate will time out at 964.048058s 2024-03-29T10:44:01.801 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:01.818+0000 7f317dfc1640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f31780af8e0 0x7f3178136660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.88:49123/0 (socket says 172.21.15.88:49123) 2024-03-29T10:44:01.801 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:01.818+0000 7f317dfc1640 1 -- 172.21.15.88:0/832891229 learned_addr learned my addr 172.21.15.88:0/832891229 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-29T10:44:01.801 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:01.818+0000 7f316e7fc640 10 client.?.objecter ms_handle_connect 0x7f31780b0860 2024-03-29T10:44:01.801 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:01.818+0000 7f316e7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:44:01.805 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:01.822+0000 7f316e7fc640 10 client.?.objecter ms_handle_connect 0x7f31780af8e0 2024-03-29T10:44:01.805 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:01.822+0000 7f316e7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:44:01.805 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:01.822+0000 7f316e7fc640 1 -- 172.21.15.88:0/832891229 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1660060104 0 0) 0x7f3178137300 con 0x7f31780b0860 2024-03-29T10:44:01.805 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:01.822+0000 7f316e7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:44:01.806 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:01.822+0000 7f316e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:01.806 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:01.822+0000 7f316e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:01.806 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:01.822+0000 7f316e7fc640 10 monclient(hunting): my global_id is 4367 2024-03-29T10:44:01.806 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:01.822+0000 7f316e7fc640 1 -- 172.21.15.88:0/832891229 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f315c004eb0 con 0x7f31780b0860 2024-03-29T10:44:01.806 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:01.822+0000 7f316e7fc640 1 -- 172.21.15.88:0/832891229 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3499948227 0 0) 0x7f3178138490 con 0x7f31780af8e0 2024-03-29T10:44:01.806 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:01.822+0000 7f316e7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:44:01.806 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:01.822+0000 7f316e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:01.806 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:01.822+0000 7f316e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:01.806 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:01.822+0000 7f316e7fc640 10 monclient(hunting): my global_id is 4330 2024-03-29T10:44:01.806 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:01.822+0000 7f316e7fc640 1 -- 172.21.15.88:0/832891229 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3178137300 con 0x7f31780af8e0 2024-03-29T10:44:01.807 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:01.823+0000 7f316e7fc640 1 -- 172.21.15.88:0/832891229 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2602645135 0 0) 0x7f3178137300 con 0x7f31780af8e0 2024-03-29T10:44:01.807 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:01.823+0000 7f316e7fc640 1 -- 172.21.15.88:0/832891229 --> v1:172.21.15.196:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3178138490 con 0x7f31780af8e0 2024-03-29T10:44:01.807 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:01.823+0000 7f316e7fc640 1 -- 172.21.15.88:0/832891229 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f3170001ca0 con 0x7f31780af8e0 2024-03-29T10:44:01.807 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:01.823+0000 7f316e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:01.807 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:01.823+0000 7f316e7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:44:01.807 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:01.823+0000 7f316e7fc640 10 monclient(hunting): dump: 2024-03-29T10:44:01.807 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:44:01.807 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:44:01.807 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:44:01.807 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:44:01.807 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:01.807 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:44:01.808 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:44:01.808 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:44:01.808 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:44:01.808 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:44:01.808 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:01.823+0000 7f316e7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:44:01.808 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:01.824+0000 7f316e7fc640 10 client.?.objecter ms_handle_connect 0x7f317813ef30 2024-03-29T10:44:01.808 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:01.824+0000 7f316e7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:44:01.808 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:01.824+0000 7f316e7fc640 1 -- 172.21.15.88:0/832891229 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 644589204 0 0) 0x7f315c004eb0 con 0x7f31780b0860 2024-03-29T10:44:01.808 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:01.824+0000 7f316e7fc640 1 -- 172.21.15.88:0/832891229 --> v1:172.21.15.88:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3178137300 con 0x7f31780b0860 2024-03-29T10:44:01.808 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:01.824+0000 7f316e7fc640 1 -- 172.21.15.88:0/832891229 <== mon.1 v1:172.21.15.196:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2575108626 0 0) 0x7f3178138490 con 0x7f31780af8e0 2024-03-29T10:44:01.808 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:01.824+0000 7f316e7fc640 1 -- 172.21.15.88:0/832891229 >> v1:172.21.15.88:6790/0 conn(0x7f31780b0860 legacy=0x7f31780b2b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:01.808 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:01.824+0000 7f316e7fc640 1 -- 172.21.15.88:0/832891229 >> v1:172.21.15.88:6789/0 conn(0x7f317813ef30 legacy=0x7f31781ee840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:01.808 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:01.824+0000 7f316e7fc640 10 monclient: _finish_hunting 0 2024-03-29T10:44:01.808 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:01.824+0000 7f316e7fc640 1 monclient: found mon.b 2024-03-29T10:44:01.809 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:01.824+0000 7f316e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:44:01.809 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:01.824+0000 7f316e7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T10:44:01.809 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:01.824+0000 7f316e7fc640 1 -- 172.21.15.88:0/832891229 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3178138770 con 0x7f31780af8e0 2024-03-29T10:44:01.809 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:01.824+0000 7f316e7fc640 10 monclient: _finish_auth 0 2024-03-29T10:44:01.809 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:01.824+0000 7f316e7fc640 10 monclient: _check_auth_tickets 2024-03-29T10:44:01.809 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:01.824+0000 7f316e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:44:01.810 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:01.825+0000 7f3180225640 5 monclient: authenticate success, global_id 4330 2024-03-29T10:44:01.810 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:01.825+0000 7f3180225640 10 monclient: _renew_subs 2024-03-29T10:44:01.810 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:01.825+0000 7f3180225640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T10:44:01.810 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:01.825+0000 7f3180225640 1 -- 172.21.15.88:0/832891229 --> v1:172.21.15.196:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3178136f20 con 0x7f31780af8e0 2024-03-29T10:44:01.810 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:01.825+0000 7f316e7fc640 1 -- 172.21.15.88:0/832891229 <== mon.1 v1:172.21.15.196:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3170001e00 con 0x7f31780af8e0 2024-03-29T10:44:01.810 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:01.825+0000 7f316e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:44:01.810 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:01.825+0000 7f316e7fc640 1 -- 172.21.15.88:0/832891229 <== mon.1 v1:172.21.15.196:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f3170003c20 con 0x7f31780af8e0 2024-03-29T10:44:01.810 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:01.825+0000 7f316e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:01.810 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:01.825+0000 7f316e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:44:01.811 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:01.825+0000 7f316e7fc640 10 monclient: dump: 2024-03-29T10:44:01.811 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:44:01.811 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:44:01.811 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:44:01.811 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:44:01.811 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:01.811 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:44:01.811 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:44:01.811 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:44:01.811 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:44:01.811 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:44:01.811 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:01.825+0000 7f3180225640 10 client.4330.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:44:01.811 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:01.825+0000 7f3180225640 10 monclient: _renew_subs 2024-03-29T10:44:01.811 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:01.825+0000 7f3180225640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T10:44:01.812 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:01.825+0000 7f3180225640 1 -- 172.21.15.88:0/832891229 --> v1:172.21.15.196:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3178137a00 con 0x7f31780af8e0 2024-03-29T10:44:01.812 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:01.825+0000 7f3180225640 1 librados: init done 2024-03-29T10:44:01.812 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:01.825+0000 7f316ffff640 4 set_mon_vals no callback set 2024-03-29T10:44:01.812 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:01.827+0000 7f3180225640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:44:01.812 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:01.827+0000 7f3180225640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:44:01.812 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:01.827+0000 7f3180225640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T10:44:01.812 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:01.827+0000 7f3180225640 1 -- 172.21.15.88:0/832891229 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3178075f10 con 0x7f31780af8e0 2024-03-29T10:44:01.812 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:01.828+0000 7f316e7fc640 1 -- 172.21.15.88:0/832891229 <== mon.1 v1:172.21.15.196:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4235629220 0 0) 0x7f3170003d80 con 0x7f31780af8e0 2024-03-29T10:44:01.814 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:01.830+0000 7f316e7fc640 1 -- 172.21.15.88:0/832891229 <== mon.1 v1:172.21.15.196:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4257673582 0 0) 0x7f3170084ca0 con 0x7f31780af8e0 2024-03-29T10:44:01.814 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:01.830+0000 7f316e7fc640 10 client.4330.objecter ms_dispatch 0x7f3178000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:44:01.814 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:01.830+0000 7f316e7fc640 3 client.4330.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:44:01.814 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:01.830+0000 7f316e7fc640 3 client.4330.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:44:01.814 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:01.831+0000 7f316e7fc640 20 client.4330.objecter dump_active .. 0 homeless 2024-03-29T10:44:01.814 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:01.831+0000 7f317d7c0640 10 monclient: get_auth_request con 0x7f315c06dec0 auth_method 0 2024-03-29T10:44:01.814 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:01.831+0000 7f316e7fc640 10 client.4330.objecter ms_handle_connect 0x7f315c06dec0 2024-03-29T10:44:01.819 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:01.836+0000 7f316e7fc640 1 -- 172.21.15.88:0/832891229 <== mon.1 v1:172.21.15.196:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f3170050580 con 0x7f31780af8e0 2024-03-29T10:44:01.819 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:01.836+0000 7f316e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:44:01.819 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:01.836+0000 7f316e7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:44:02.134 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:02.151+0000 7f3180225640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:44:02.134 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:02.151+0000 7f3180225640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:44:02.134 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:02.151+0000 7f3180225640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T10:44:02.134 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:02.151+0000 7f3180225640 1 -- 172.21.15.88:0/832891229 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3178072330 con 0x7f31780af8e0 2024-03-29T10:44:02.137 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:02.154+0000 7f316e7fc640 1 -- 172.21.15.88:0/832891229 <== mon.1 v1:172.21.15.196:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 344506374) 0x7f31700524b0 con 0x7f31780af8e0 2024-03-29T10:44:02.137 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:02.154+0000 7f316e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:44:02.137 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:02.154+0000 7f316e7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:44:02.138 INFO:teuthology.misc.health.smithi088.stdout: 2024-03-29T10:44:02.138 INFO:teuthology.misc.health.smithi088.stdout:{"epoch":10,"fsid":"869f5188-f92b-42dd-a46d-8c646e0388a9","created":"2024-03-29T10:40:40.285087+0000","modified":"2024-03-29T10:40:42.536230+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:40:42.536230+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a26c0c1c-8de0-438c-a3f6-3df10ea61574","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5de24915-8dec-49b6-8b16-f3170197c591","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ac054af3-d589-4565-9644-e6af75975c53","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3ec4f045-c714-4a22-bd1a-e07330b45ab3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"cddb9867-8e30-4951-9b13-3d1e78b9c45a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dc6d9911-785d-432f-8b53-56a1ccab302f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"77c88437-b587-496e-8b4d-1824ea8b5305","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ce963d96-5a8e-44c3-99de-c0dbebe234ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:44:02.143 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:02.158+0000 7f3180225640 10 librados: watch_flush enter 2024-03-29T10:44:02.143 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:02.158+0000 7f3180225640 10 librados: watch_flush exit 2024-03-29T10:44:02.143 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:02.158+0000 7f3180225640 20 client.4330.objecter shutdown clearing up homeless session... 2024-03-29T10:44:02.143 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:02.158+0000 7f3180225640 10 client.4330.objecter successfully canceled tick 2024-03-29T10:44:02.143 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:02.159+0000 7f3180225640 1 -- 172.21.15.88:0/832891229 >> v1:172.21.15.88:6816/1752861349 conn(0x7f315c06dec0 legacy=0x7f315c070380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:02.143 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:02.159+0000 7f3180225640 10 monclient: shutdown 2024-03-29T10:44:02.143 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:02.159+0000 7f3180225640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:44:02.143 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:02.159+0000 7f3180225640 1 -- 172.21.15.88:0/832891229 >> v1:172.21.15.196:6789/0 conn(0x7f31780af8e0 legacy=0x7f3178136660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:02.144 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:02.159+0000 7f3180225640 1 -- 172.21.15.88:0/832891229 shutdown_connections 2024-03-29T10:44:02.144 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:02.159+0000 7f3180225640 1 -- 172.21.15.88:0/832891229 >> 172.21.15.88:0/832891229 conn(0x7f317806b660 msgr2=0x7f317806ba40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:44:02.144 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:02.160+0000 7f3180225640 1 -- 172.21.15.88:0/832891229 shutdown_connections 2024-03-29T10:44:02.144 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:02.160+0000 7f3180225640 1 -- 172.21.15.88:0/832891229 wait complete. 2024-03-29T10:44:02.144 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:02.160+0000 7f3180225640 1 librados: shutdown 2024-03-29T10:44:02.159 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:44:08.967 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:44:09.111 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:09.128+0000 7f9beecff640 10 monclient: get_monmap_and_config 2024-03-29T10:44:09.111 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:09.128+0000 7f9beecff640 10 monclient: build_initial_monmap 2024-03-29T10:44:09.112 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:09.128+0000 7f9beecff640 10 monclient: monmap: 2024-03-29T10:44:09.112 INFO:teuthology.misc.health.smithi088.stderr:epoch 0 2024-03-29T10:44:09.112 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:44:09.112 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:44:09.129556+0000 2024-03-29T10:44:09.112 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:44:09.129556+0000 2024-03-29T10:44:09.112 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 0 (unknown) 2024-03-29T10:44:09.112 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:44:09.112 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-03-29T10:44:09.112 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-29T10:44:09.112 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-b 2024-03-29T10:44:09.112 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:44:09.114 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:09.129+0000 7f9beecff640 1 Processor -- start 2024-03-29T10:44:09.114 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:09.129+0000 7f9beecff640 1 -- start start 2024-03-29T10:44:09.114 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:09.129+0000 7f9beecff640 10 monclient: init 2024-03-29T10:44:09.114 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:09.130+0000 7f9beecff640 10 monclient: _reopen_session rank -1 2024-03-29T10:44:09.114 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:09.130+0000 7f9beecff640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:44:09.114 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:09.130+0000 7f9beecff640 10 monclient(hunting): picked mon.noname-b con 0x7f9be81422b0 addr v1:172.21.15.88:6790/0 2024-03-29T10:44:09.115 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:09.130+0000 7f9beecff640 10 monclient(hunting): picked mon.noname-a con 0x7f9be8142690 addr v1:172.21.15.88:6789/0 2024-03-29T10:44:09.115 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:09.130+0000 7f9beecff640 10 monclient(hunting): picked mon.noname-c con 0x7f9be80b17d0 addr v1:172.21.15.196:6789/0 2024-03-29T10:44:09.115 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:09.130+0000 7f9be7fff640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f9be8142690 0x7f9be80b2590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:35518/0 (socket says 172.21.15.88:35518) 2024-03-29T10:44:09.115 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:09.130+0000 7f9beca9b640 1 --1- >> v1:172.21.15.88:6790/0 conn(0x7f9be81422b0 0x7f9be813f4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6790/0 says I am v1:172.21.15.88:40062/0 (socket says 172.21.15.88:40062) 2024-03-29T10:44:09.115 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:09.130+0000 7f9be7fff640 1 -- 172.21.15.88:0/4193391109 learned_addr learned my addr 172.21.15.88:0/4193391109 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-29T10:44:09.115 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:09.131+0000 7f9beecff640 1 -- 172.21.15.88:0/4193391109 --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9be8142c80 con 0x7f9be8142690 2024-03-29T10:44:09.115 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:09.131+0000 7f9beecff640 1 -- 172.21.15.88:0/4193391109 --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9be80aae40 con 0x7f9be80b17d0 2024-03-29T10:44:09.115 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:09.131+0000 7f9beecff640 1 -- 172.21.15.88:0/4193391109 --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9be813fbd0 con 0x7f9be81422b0 2024-03-29T10:44:09.115 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:09.131+0000 7f9beecff640 10 monclient(hunting): _renew_subs 2024-03-29T10:44:09.115 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:09.131+0000 7f9beecff640 10 monclient(hunting): authenticate will time out at 971.367349s 2024-03-29T10:44:09.128 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:09.132+0000 7f9be77fe640 1 -- 172.21.15.88:0/4193391109 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 480951821 0 0) 0x7f9be80aae40 con 0x7f9be80b17d0 2024-03-29T10:44:09.128 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:09.132+0000 7f9be77fe640 10 monclient(hunting): _negotiate 2024-03-29T10:44:09.128 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:09.132+0000 7f9be77fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:09.128 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:09.132+0000 7f9be77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:09.128 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:09.132+0000 7f9be77fe640 10 monclient(hunting): my global_id is 4333 2024-03-29T10:44:09.128 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:09.132+0000 7f9be77fe640 1 -- 172.21.15.88:0/4193391109 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9bd4003520 con 0x7f9be80b17d0 2024-03-29T10:44:09.128 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:09.132+0000 7f9be77fe640 1 -- 172.21.15.88:0/4193391109 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4012482309 0 0) 0x7f9be813fbd0 con 0x7f9be81422b0 2024-03-29T10:44:09.129 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:09.132+0000 7f9be77fe640 10 monclient(hunting): _negotiate 2024-03-29T10:44:09.129 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:09.132+0000 7f9be77fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:09.129 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:09.132+0000 7f9be77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:09.129 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:09.132+0000 7f9be77fe640 10 monclient(hunting): my global_id is 4370 2024-03-29T10:44:09.129 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:09.132+0000 7f9be77fe640 1 -- 172.21.15.88:0/4193391109 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9be80aae40 con 0x7f9be81422b0 2024-03-29T10:44:09.129 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:09.132+0000 7f9be77fe640 1 -- 172.21.15.88:0/4193391109 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2999964747 0 0) 0x7f9be8142c80 con 0x7f9be8142690 2024-03-29T10:44:09.129 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:09.132+0000 7f9be77fe640 10 monclient(hunting): _negotiate 2024-03-29T10:44:09.129 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:09.132+0000 7f9be77fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:09.129 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:09.132+0000 7f9be77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:09.129 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:09.132+0000 7f9be77fe640 10 monclient(hunting): my global_id is 4338 2024-03-29T10:44:09.129 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:09.132+0000 7f9be77fe640 1 -- 172.21.15.88:0/4193391109 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9be813fbd0 con 0x7f9be8142690 2024-03-29T10:44:09.129 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:09.132+0000 7f9be77fe640 1 -- 172.21.15.88:0/4193391109 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 57641668 0 0) 0x7f9bd4003520 con 0x7f9be80b17d0 2024-03-29T10:44:09.129 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:09.132+0000 7f9be77fe640 1 -- 172.21.15.88:0/4193391109 >> v1:172.21.15.88:6790/0 conn(0x7f9be81422b0 legacy=0x7f9be813f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:09.129 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:09.132+0000 7f9be77fe640 1 -- 172.21.15.88:0/4193391109 >> v1:172.21.15.88:6789/0 conn(0x7f9be8142690 legacy=0x7f9be80b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:09.130 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:09.132+0000 7f9be77fe640 10 monclient: _finish_hunting 0 2024-03-29T10:44:09.130 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:09.132+0000 7f9be77fe640 1 monclient: found mon.noname-c 2024-03-29T10:44:09.130 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:09.132+0000 7f9be77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:44:09.130 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:09.132+0000 7f9be77fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.196:6789/0 2024-03-29T10:44:09.130 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:09.132+0000 7f9be77fe640 1 -- 172.21.15.88:0/4193391109 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9be8150930 con 0x7f9be80b17d0 2024-03-29T10:44:09.130 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:09.132+0000 7f9be77fe640 10 monclient: _finish_auth 0 2024-03-29T10:44:09.130 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:09.132+0000 7f9be77fe640 10 monclient: _check_auth_tickets 2024-03-29T10:44:09.130 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:09.132+0000 7f9be77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:44:09.130 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:09.132+0000 7f9be77fe640 1 -- 172.21.15.88:0/4193391109 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f9bd8001a10 con 0x7f9be80b17d0 2024-03-29T10:44:09.130 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:09.132+0000 7f9be77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:09.130 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:09.132+0000 7f9be77fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:44:09.130 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:09.132+0000 7f9be77fe640 10 monclient: dump: 2024-03-29T10:44:09.130 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:44:09.130 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:44:09.131 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:44:09.131 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:44:09.131 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:09.131 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:44:09.131 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:44:09.131 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:44:09.131 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:44:09.131 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:44:09.131 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:09.132+0000 7f9beecff640 5 monclient: authenticate success, global_id 4333 2024-03-29T10:44:09.131 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:09.132+0000 7f9beecff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:44:09.131 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:09.132+0000 7f9be77fe640 1 -- 172.21.15.88:0/4193391109 <== mon.1 v1:172.21.15.196:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9bd8001b70 con 0x7f9be80b17d0 2024-03-29T10:44:09.131 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:09.132+0000 7f9be77fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:44:09.131 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:09.132+0000 7f9beecff640 10 monclient: get_monmap_and_config success 2024-03-29T10:44:09.131 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:09.132+0000 7f9beecff640 4 set_mon_vals no callback set 2024-03-29T10:44:09.132 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:09.133+0000 7f9be77fe640 1 -- 172.21.15.88:0/4193391109 <== mon.1 v1:172.21.15.196:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f9bd8001f00 con 0x7f9be80b17d0 2024-03-29T10:44:09.132 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:09.133+0000 7f9be77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:09.132 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:09.133+0000 7f9be77fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:44:09.132 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:09.133+0000 7f9be77fe640 10 monclient: dump: 2024-03-29T10:44:09.132 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:44:09.132 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:44:09.132 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:44:09.132 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:44:09.132 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:09.132 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:44:09.132 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:44:09.132 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:44:09.132 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:44:09.132 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:44:09.133 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:09.133+0000 7f9beecff640 10 monclient: shutdown 2024-03-29T10:44:09.133 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:09.133+0000 7f9beecff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:44:09.133 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:09.133+0000 7f9beecff640 1 -- 172.21.15.88:0/4193391109 >> v1:172.21.15.196:6789/0 conn(0x7f9be80b17d0 legacy=0x7f9be80b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:09.133 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:09.133+0000 7f9beecff640 1 -- 172.21.15.88:0/4193391109 shutdown_connections 2024-03-29T10:44:09.133 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:09.133+0000 7f9beecff640 1 -- 172.21.15.88:0/4193391109 >> 172.21.15.88:0/4193391109 conn(0x7f9be8066a00 msgr2=0x7f9be8066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:44:09.133 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:09.134+0000 7f9beecff640 1 -- 172.21.15.88:0/4193391109 shutdown_connections 2024-03-29T10:44:09.133 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:09.134+0000 7f9beecff640 1 -- 172.21.15.88:0/4193391109 wait complete. 2024-03-29T10:44:09.133 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:09.135+0000 7f9beecff640 10 monclient: build_initial_monmap 2024-03-29T10:44:09.133 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:09.135+0000 7f9beecff640 10 monclient: monmap: 2024-03-29T10:44:09.133 INFO:teuthology.misc.health.smithi088.stderr:epoch 0 2024-03-29T10:44:09.133 INFO:teuthology.misc.health.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:44:09.133 INFO:teuthology.misc.health.smithi088.stderr:last_changed 0.000000 2024-03-29T10:44:09.133 INFO:teuthology.misc.health.smithi088.stderr:created 0.000000 2024-03-29T10:44:09.133 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 0 (unknown) 2024-03-29T10:44:09.133 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:44:09.133 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-03-29T10:44:09.134 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-29T10:44:09.134 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-c 2024-03-29T10:44:09.134 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:44:09.134 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:09.135+0000 7f9beecff640 1 librados: starting msgr at 2024-03-29T10:44:09.134 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:09.135+0000 7f9beecff640 1 librados: starting objecter 2024-03-29T10:44:09.134 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:09.135+0000 7f9beecff640 1 Processor -- start 2024-03-29T10:44:09.134 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:09.135+0000 7f9beecff640 1 -- start start 2024-03-29T10:44:09.134 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:09.135+0000 7f9beecff640 1 librados: setting wanted keys 2024-03-29T10:44:09.134 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:09.135+0000 7f9beecff640 1 librados: calling monclient init 2024-03-29T10:44:09.134 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:09.135+0000 7f9beecff640 10 monclient: init 2024-03-29T10:44:09.134 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:09.136+0000 7f9beecff640 10 monclient: _reopen_session rank -1 2024-03-29T10:44:09.134 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:09.136+0000 7f9beecff640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:44:09.134 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:09.137+0000 7f9beecff640 10 monclient(hunting): picked mon.noname-b con 0x7f9be80b17d0 addr v1:172.21.15.196:6789/0 2024-03-29T10:44:09.134 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:09.137+0000 7f9beecff640 10 monclient(hunting): picked mon.noname-a con 0x7f9be81422b0 addr v1:172.21.15.88:6789/0 2024-03-29T10:44:09.135 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:09.137+0000 7f9beca9b640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f9be80b17d0 0x7f9be806f130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.88:54988/0 (socket says 172.21.15.88:54988) 2024-03-29T10:44:09.135 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:09.137+0000 7f9be7fff640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f9be81422b0 0x7f9be80712e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:35728/0 (socket says 172.21.15.88:35728) 2024-03-29T10:44:09.135 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:09.137+0000 7f9bed29c640 1 --1- >> v1:172.21.15.88:6790/0 conn(0x7f9be8142690 0x7f9be80719f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6790/0 says I am v1:172.21.15.88:40390/0 (socket says 172.21.15.88:40390) 2024-03-29T10:44:09.135 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:09.137+0000 7f9beca9b640 1 -- 172.21.15.88:0/2063495829 learned_addr learned my addr 172.21.15.88:0/2063495829 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-29T10:44:09.135 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:09.137+0000 7f9beecff640 10 monclient(hunting): picked mon.noname-c con 0x7f9be8142690 addr v1:172.21.15.88:6790/0 2024-03-29T10:44:09.135 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:09.137+0000 7f9be57fa640 10 client.?.objecter ms_handle_connect 0x7f9be8142690 2024-03-29T10:44:09.135 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:09.137+0000 7f9be57fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:44:09.135 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:09.137+0000 7f9be57fa640 10 client.?.objecter ms_handle_connect 0x7f9be81422b0 2024-03-29T10:44:09.135 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:09.137+0000 7f9be57fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:44:09.135 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:09.137+0000 7f9be57fa640 10 client.?.objecter ms_handle_connect 0x7f9be80b17d0 2024-03-29T10:44:09.135 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:09.137+0000 7f9be57fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:44:09.135 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:09.137+0000 7f9beecff640 1 -- 172.21.15.88:0/2063495829 --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9be8070e00 con 0x7f9be81422b0 2024-03-29T10:44:09.135 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:09.137+0000 7f9beecff640 1 -- 172.21.15.88:0/2063495829 --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9be8070700 con 0x7f9be80b17d0 2024-03-29T10:44:09.136 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:09.137+0000 7f9beecff640 1 -- 172.21.15.88:0/2063495829 --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9be80709e0 con 0x7f9be8142690 2024-03-29T10:44:09.136 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:09.137+0000 7f9beecff640 10 monclient(hunting): _renew_subs 2024-03-29T10:44:09.136 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:09.137+0000 7f9beecff640 10 monclient(hunting): authenticate will time out at 971.373847s 2024-03-29T10:44:09.136 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:09.138+0000 7f9be57fa640 1 -- 172.21.15.88:0/2063495829 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2433737554 0 0) 0x7f9be80709e0 con 0x7f9be8142690 2024-03-29T10:44:09.136 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:09.138+0000 7f9be57fa640 10 monclient(hunting): _negotiate 2024-03-29T10:44:09.136 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:09.138+0000 7f9be57fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:09.136 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:09.138+0000 7f9be57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:09.136 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:09.138+0000 7f9be57fa640 10 monclient(hunting): my global_id is 4373 2024-03-29T10:44:09.136 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:09.138+0000 7f9be57fa640 1 -- 172.21.15.88:0/2063495829 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9bc8004eb0 con 0x7f9be8142690 2024-03-29T10:44:09.136 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:09.138+0000 7f9be57fa640 1 -- 172.21.15.88:0/2063495829 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1390820474 0 0) 0x7f9be8070700 con 0x7f9be80b17d0 2024-03-29T10:44:09.136 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:09.138+0000 7f9be57fa640 10 monclient(hunting): _negotiate 2024-03-29T10:44:09.136 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:09.138+0000 7f9be57fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:09.136 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:09.138+0000 7f9be57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:09.136 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:09.138+0000 7f9be57fa640 10 monclient(hunting): my global_id is 4336 2024-03-29T10:44:09.137 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:09.138+0000 7f9be57fa640 1 -- 172.21.15.88:0/2063495829 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9be80709e0 con 0x7f9be80b17d0 2024-03-29T10:44:09.137 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:09.138+0000 7f9be57fa640 1 -- 172.21.15.88:0/2063495829 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 32790675 0 0) 0x7f9be8070e00 con 0x7f9be81422b0 2024-03-29T10:44:09.137 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:09.138+0000 7f9be57fa640 10 monclient(hunting): _negotiate 2024-03-29T10:44:09.137 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:09.138+0000 7f9be57fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:09.137 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:09.138+0000 7f9be57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:09.137 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:09.138+0000 7f9be57fa640 10 monclient(hunting): my global_id is 4341 2024-03-29T10:44:09.137 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:09.138+0000 7f9be57fa640 1 -- 172.21.15.88:0/2063495829 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9be8070700 con 0x7f9be81422b0 2024-03-29T10:44:09.137 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:09.138+0000 7f9be57fa640 1 -- 172.21.15.88:0/2063495829 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2262896680 0 0) 0x7f9bc8004eb0 con 0x7f9be8142690 2024-03-29T10:44:09.137 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:09.138+0000 7f9be57fa640 1 -- 172.21.15.88:0/2063495829 --> v1:172.21.15.88:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9be8070e00 con 0x7f9be8142690 2024-03-29T10:44:09.137 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:09.138+0000 7f9be57fa640 1 -- 172.21.15.88:0/2063495829 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f9bd80018d0 con 0x7f9be8142690 2024-03-29T10:44:09.137 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:09.138+0000 7f9be57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:09.137 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:09.138+0000 7f9be57fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:44:09.137 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:09.138+0000 7f9be57fa640 10 monclient(hunting): dump: 2024-03-29T10:44:09.137 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:44:09.138 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:44:09.138 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:44:09.138 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:44:09.138 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:09.138 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:44:09.138 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:44:09.138 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:44:09.138 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:44:09.138 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:44:09.138 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:09.138+0000 7f9be57fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:44:09.138 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:09.138+0000 7f9be57fa640 1 -- 172.21.15.88:0/2063495829 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3754551135 0 0) 0x7f9be8070700 con 0x7f9be81422b0 2024-03-29T10:44:09.138 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:09.139+0000 7f9be57fa640 1 -- 172.21.15.88:0/2063495829 --> v1:172.21.15.88:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9bc8004eb0 con 0x7f9be81422b0 2024-03-29T10:44:09.138 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:09.139+0000 7f9be57fa640 1 -- 172.21.15.88:0/2063495829 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f9bd00017d0 con 0x7f9be81422b0 2024-03-29T10:44:09.138 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:09.139+0000 7f9be57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:09.139 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:09.139+0000 7f9be57fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:44:09.139 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:09.139+0000 7f9be57fa640 10 monclient(hunting): dump: 2024-03-29T10:44:09.139 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:44:09.139 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:44:09.139 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:44:09.139 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:44:09.139 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:09.139 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:44:09.139 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:44:09.139 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:44:09.139 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:44:09.139 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:44:09.139 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:09.139+0000 7f9be57fa640 1 -- 172.21.15.88:0/2063495829 <== mon.2 v1:172.21.15.88:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4208111100 0 0) 0x7f9be8070e00 con 0x7f9be8142690 2024-03-29T10:44:09.139 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:09.139+0000 7f9be57fa640 1 -- 172.21.15.88:0/2063495829 >> v1:172.21.15.196:6789/0 conn(0x7f9be80b17d0 legacy=0x7f9be806f130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:09.140 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:09.139+0000 7f9be57fa640 1 -- 172.21.15.88:0/2063495829 >> v1:172.21.15.88:6789/0 conn(0x7f9be81422b0 legacy=0x7f9be80712e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:09.140 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:09.139+0000 7f9be57fa640 10 monclient: _finish_hunting 0 2024-03-29T10:44:09.140 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:09.139+0000 7f9be57fa640 1 monclient: found mon.c 2024-03-29T10:44:09.140 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:09.139+0000 7f9be57fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:44:09.140 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:09.139+0000 7f9be57fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-03-29T10:44:09.140 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:09.139+0000 7f9be57fa640 1 -- 172.21.15.88:0/2063495829 --> v1:172.21.15.88:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9be806e310 con 0x7f9be8142690 2024-03-29T10:44:09.140 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:09.139+0000 7f9be57fa640 10 monclient: _finish_auth 0 2024-03-29T10:44:09.140 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:09.139+0000 7f9be57fa640 10 monclient: _check_auth_tickets 2024-03-29T10:44:09.140 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:09.139+0000 7f9be57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:44:09.140 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:09.139+0000 7f9beecff640 5 monclient: authenticate success, global_id 4373 2024-03-29T10:44:09.140 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:09.139+0000 7f9beecff640 10 monclient: _renew_subs 2024-03-29T10:44:09.140 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:09.139+0000 7f9beecff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-03-29T10:44:09.140 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:09.139+0000 7f9beecff640 1 -- 172.21.15.88:0/2063495829 --> v1:172.21.15.88:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9be81e7210 con 0x7f9be8142690 2024-03-29T10:44:09.140 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:09.139+0000 7f9beecff640 10 client.4373.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:44:09.141 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:09.139+0000 7f9beecff640 10 monclient: _renew_subs 2024-03-29T10:44:09.141 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:09.139+0000 7f9beecff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-03-29T10:44:09.141 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:09.139+0000 7f9beecff640 1 -- 172.21.15.88:0/2063495829 --> v1:172.21.15.88:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9be81e7580 con 0x7f9be8142690 2024-03-29T10:44:09.141 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:09.139+0000 7f9beecff640 1 librados: init done 2024-03-29T10:44:09.141 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:09.140+0000 7f9be57fa640 1 -- 172.21.15.88:0/2063495829 <== mon.2 v1:172.21.15.88:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9bd80014e0 con 0x7f9be8142690 2024-03-29T10:44:09.141 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:09.140+0000 7f9be57fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:44:09.141 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:09.140+0000 7f9be67fc640 4 set_mon_vals no callback set 2024-03-29T10:44:09.141 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:09.140+0000 7f9beecff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:44:09.141 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:09.141+0000 7f9be57fa640 1 -- 172.21.15.88:0/2063495829 <== mon.2 v1:172.21.15.88:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f9bd8003dc0 con 0x7f9be8142690 2024-03-29T10:44:09.141 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:09.141+0000 7f9beecff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:44:09.141 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:09.141+0000 7f9beecff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-03-29T10:44:09.141 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:09.141+0000 7f9beecff640 1 -- 172.21.15.88:0/2063495829 --> v1:172.21.15.88:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9bcc005080 con 0x7f9be8142690 2024-03-29T10:44:09.141 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:09.141+0000 7f9be57fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:09.141 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:09.141+0000 7f9be57fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:44:09.141 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:09.141+0000 7f9be57fa640 10 monclient: dump: 2024-03-29T10:44:09.142 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:44:09.142 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:44:09.142 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:44:09.142 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:44:09.142 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:09.142 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:44:09.142 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:44:09.142 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:44:09.142 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:44:09.142 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:44:09.142 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:09.142+0000 7f9be57fa640 1 -- 172.21.15.88:0/2063495829 <== mon.2 v1:172.21.15.88:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4235629220 0 0) 0x7f9bd8003f20 con 0x7f9be8142690 2024-03-29T10:44:09.142 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:09.143+0000 7f9beca9b640 10 monclient: get_auth_request con 0x7f9bc806df70 auth_method 0 2024-03-29T10:44:09.142 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:09.143+0000 7f9be57fa640 1 -- 172.21.15.88:0/2063495829 <== mon.2 v1:172.21.15.88:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4257673582 0 0) 0x7f9bd8084d40 con 0x7f9be8142690 2024-03-29T10:44:09.143 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:09.143+0000 7f9be57fa640 10 client.4373.objecter ms_dispatch 0x7f9be8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:44:09.143 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:09.143+0000 7f9be57fa640 3 client.4373.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:44:09.143 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:09.143+0000 7f9be57fa640 3 client.4373.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:44:09.143 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:09.143+0000 7f9be57fa640 20 client.4373.objecter dump_active .. 0 homeless 2024-03-29T10:44:09.143 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:09.143+0000 7f9be57fa640 10 client.4373.objecter ms_handle_connect 0x7f9bc806df70 2024-03-29T10:44:09.143 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:09.153+0000 7f9be57fa640 1 -- 172.21.15.88:0/2063495829 <== mon.2 v1:172.21.15.88:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9bd8050620 con 0x7f9be8142690 2024-03-29T10:44:09.143 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:09.153+0000 7f9be57fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:44:09.143 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:09.153+0000 7f9be57fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:44:09.489 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:09.506+0000 7f9beecff640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:44:09.489 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:09.506+0000 7f9beecff640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:44:09.489 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:09.506+0000 7f9beecff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-03-29T10:44:09.489 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:09.506+0000 7f9beecff640 1 -- 172.21.15.88:0/2063495829 --> v1:172.21.15.88:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9bcc005330 con 0x7f9be8142690 2024-03-29T10:44:09.490 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:09.507+0000 7f9be57fa640 1 -- 172.21.15.88:0/2063495829 <== mon.2 v1:172.21.15.88:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 344506374) 0x7f9bd8052550 con 0x7f9be8142690 2024-03-29T10:44:09.490 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:09.507+0000 7f9be57fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:44:09.490 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:09.507+0000 7f9be57fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:44:09.490 INFO:teuthology.misc.health.smithi088.stdout: 2024-03-29T10:44:09.490 INFO:teuthology.misc.health.smithi088.stdout:{"epoch":10,"fsid":"869f5188-f92b-42dd-a46d-8c646e0388a9","created":"2024-03-29T10:40:40.285087+0000","modified":"2024-03-29T10:40:42.536230+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:40:42.536230+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a26c0c1c-8de0-438c-a3f6-3df10ea61574","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5de24915-8dec-49b6-8b16-f3170197c591","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ac054af3-d589-4565-9644-e6af75975c53","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3ec4f045-c714-4a22-bd1a-e07330b45ab3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"cddb9867-8e30-4951-9b13-3d1e78b9c45a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dc6d9911-785d-432f-8b53-56a1ccab302f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"77c88437-b587-496e-8b4d-1824ea8b5305","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ce963d96-5a8e-44c3-99de-c0dbebe234ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:44:09.494 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:09.510+0000 7f9beecff640 10 librados: watch_flush enter 2024-03-29T10:44:09.494 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:09.510+0000 7f9beecff640 10 librados: watch_flush exit 2024-03-29T10:44:09.494 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:09.510+0000 7f9beecff640 20 client.4373.objecter shutdown clearing up homeless session... 2024-03-29T10:44:09.494 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:09.510+0000 7f9beecff640 10 client.4373.objecter successfully canceled tick 2024-03-29T10:44:09.494 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:09.510+0000 7f9beecff640 1 -- 172.21.15.88:0/2063495829 >> v1:172.21.15.88:6816/1752861349 conn(0x7f9bc806df70 legacy=0x7f9bc8070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:09.494 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:09.510+0000 7f9beecff640 10 monclient: shutdown 2024-03-29T10:44:09.494 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:09.510+0000 7f9beecff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:44:09.494 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:09.510+0000 7f9beecff640 1 -- 172.21.15.88:0/2063495829 >> v1:172.21.15.88:6790/0 conn(0x7f9be8142690 legacy=0x7f9be80719f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:09.494 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:09.511+0000 7f9beecff640 1 -- 172.21.15.88:0/2063495829 shutdown_connections 2024-03-29T10:44:09.494 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:09.511+0000 7f9beecff640 1 -- 172.21.15.88:0/2063495829 >> 172.21.15.88:0/2063495829 conn(0x7f9be8066a00 msgr2=0x7f9be814f490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:44:09.494 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:09.511+0000 7f9beecff640 1 -- 172.21.15.88:0/2063495829 shutdown_connections 2024-03-29T10:44:09.495 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:09.511+0000 7f9beecff640 1 -- 172.21.15.88:0/2063495829 wait complete. 2024-03-29T10:44:09.495 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:09.511+0000 7f9beecff640 1 librados: shutdown 2024-03-29T10:44:09.508 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:44:16.315 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:44:16.465 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:16.481+0000 7f90d134b640 10 monclient: get_monmap_and_config 2024-03-29T10:44:16.465 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:16.481+0000 7f90d134b640 10 monclient: build_initial_monmap 2024-03-29T10:44:16.465 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:16.481+0000 7f90d134b640 10 monclient: monmap: 2024-03-29T10:44:16.465 INFO:teuthology.misc.health.smithi088.stderr:epoch 0 2024-03-29T10:44:16.465 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:44:16.465 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:44:16.482892+0000 2024-03-29T10:44:16.465 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:44:16.482892+0000 2024-03-29T10:44:16.465 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 0 (unknown) 2024-03-29T10:44:16.465 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:44:16.465 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-03-29T10:44:16.465 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-29T10:44:16.465 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-b 2024-03-29T10:44:16.465 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:44:16.468 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:16.485+0000 7f90d134b640 1 Processor -- start 2024-03-29T10:44:16.468 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:16.485+0000 7f90d134b640 1 -- start start 2024-03-29T10:44:16.468 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:16.485+0000 7f90d134b640 10 monclient: init 2024-03-29T10:44:16.469 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:16.486+0000 7f90d134b640 10 monclient: _reopen_session rank -1 2024-03-29T10:44:16.469 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:16.486+0000 7f90d134b640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:44:16.469 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:16.486+0000 7f90d134b640 10 monclient(hunting): picked mon.noname-a con 0x7f90cc0b1f60 addr v1:172.21.15.88:6789/0 2024-03-29T10:44:16.469 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:16.486+0000 7f90d134b640 10 monclient(hunting): picked mon.noname-b con 0x7f90cc071eb0 addr v1:172.21.15.88:6790/0 2024-03-29T10:44:16.469 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:16.486+0000 7f90d134b640 10 monclient(hunting): picked mon.noname-c con 0x7f90cc143d10 addr v1:172.21.15.196:6789/0 2024-03-29T10:44:16.469 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:16.486+0000 7f90d134b640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90cc06ef80 con 0x7f90cc0b1f60 2024-03-29T10:44:16.470 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:16.486+0000 7f90d134b640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90cc06f260 con 0x7f90cc143d10 2024-03-29T10:44:16.470 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:16.486+0000 7f90d134b640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90cc0b40c0 con 0x7f90cc071eb0 2024-03-29T10:44:16.470 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:16.486+0000 7f90d134b640 10 monclient(hunting): _renew_subs 2024-03-29T10:44:16.470 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:16.486+0000 7f90d134b640 10 monclient(hunting): authenticate will time out at 978.722346s 2024-03-29T10:44:16.470 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:16.486+0000 7f90caffd640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f90cc0b1f60 0x7f90cc143600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:50784/0 (socket says 172.21.15.88:50784) 2024-03-29T10:44:16.470 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:16.486+0000 7f90caffd640 1 -- 172.21.15.88:0/2536790102 learned_addr learned my addr 172.21.15.88:0/2536790102 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-29T10:44:16.472 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:16.488+0000 7f90ca7fc640 1 -- 172.21.15.88:0/2536790102 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1931680246 0 0) 0x7f90cc06ef80 con 0x7f90cc0b1f60 2024-03-29T10:44:16.472 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:16.488+0000 7f90ca7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:44:16.472 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:16.488+0000 7f90ca7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:16.472 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:16.488+0000 7f90ca7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:16.472 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:16.488+0000 7f90ca7fc640 10 monclient(hunting): my global_id is 4344 2024-03-29T10:44:16.472 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:16.488+0000 7f90ca7fc640 1 -- 172.21.15.88:0/2536790102 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f90b8003520 con 0x7f90cc0b1f60 2024-03-29T10:44:16.472 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:16.488+0000 7f90ca7fc640 1 -- 172.21.15.88:0/2536790102 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2899740221 0 0) 0x7f90b8003520 con 0x7f90cc0b1f60 2024-03-29T10:44:16.472 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:16.488+0000 7f90ca7fc640 1 -- 172.21.15.88:0/2536790102 >> v1:172.21.15.88:6790/0 conn(0x7f90cc071eb0 legacy=0x7f90cc146470 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:44:16.473 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:16.490+0000 7f90ca7fc640 1 -- 172.21.15.88:0/2536790102 >> v1:172.21.15.196:6789/0 conn(0x7f90cc143d10 legacy=0x7f90cc146b80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:44:16.476 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:16.493+0000 7f90ca7fc640 10 monclient: _finish_hunting 0 2024-03-29T10:44:16.476 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:16.493+0000 7f90ca7fc640 1 monclient: found mon.noname-a 2024-03-29T10:44:16.476 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:16.493+0000 7f90ca7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:44:16.476 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:16.493+0000 7f90ca7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.88:6789/0 2024-03-29T10:44:16.477 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:16.493+0000 7f90ca7fc640 1 -- 172.21.15.88:0/2536790102 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90cc155d70 con 0x7f90cc0b1f60 2024-03-29T10:44:16.477 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:16.493+0000 7f90ca7fc640 10 monclient: _finish_auth 0 2024-03-29T10:44:16.477 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:16.493+0000 7f90ca7fc640 10 monclient: _check_auth_tickets 2024-03-29T10:44:16.477 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:16.493+0000 7f90ca7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:44:16.477 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:16.493+0000 7f90d134b640 5 monclient: authenticate success, global_id 4344 2024-03-29T10:44:16.477 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:16.493+0000 7f90d134b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:44:16.477 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:16.493+0000 7f90ca7fc640 1 -- 172.21.15.88:0/2536790102 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f90bc001690 con 0x7f90cc0b1f60 2024-03-29T10:44:16.477 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:16.493+0000 7f90ca7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:16.477 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:16.493+0000 7f90ca7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:44:16.477 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:16.493+0000 7f90ca7fc640 10 monclient: dump: 2024-03-29T10:44:16.477 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:44:16.477 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:44:16.477 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:44:16.477 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:44:16.478 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:16.478 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:44:16.478 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:44:16.478 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:44:16.478 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:44:16.478 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:44:16.478 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:16.493+0000 7f90d134b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:44:16.478 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:16.494+0000 7f90ca7fc640 1 -- 172.21.15.88:0/2536790102 <== mon.0 v1:172.21.15.88:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f90bc001920 con 0x7f90cc0b1f60 2024-03-29T10:44:16.478 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:16.494+0000 7f90ca7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:44:16.478 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:16.494+0000 7f90ca7fc640 1 -- 172.21.15.88:0/2536790102 <== mon.0 v1:172.21.15.88:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f90bc001cb0 con 0x7f90cc0b1f60 2024-03-29T10:44:16.479 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:16.494+0000 7f90ca7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:16.479 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:16.494+0000 7f90ca7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:44:16.479 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:16.494+0000 7f90ca7fc640 10 monclient: dump: 2024-03-29T10:44:16.479 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:44:16.479 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:44:16.479 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:44:16.479 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:44:16.479 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:16.479 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:44:16.479 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:44:16.479 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:44:16.479 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:44:16.479 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:44:16.480 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:16.494+0000 7f90d134b640 10 monclient: get_monmap_and_config success 2024-03-29T10:44:16.480 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:16.494+0000 7f90d134b640 4 set_mon_vals no callback set 2024-03-29T10:44:16.480 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:16.495+0000 7f90d134b640 10 monclient: shutdown 2024-03-29T10:44:16.480 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:16.495+0000 7f90d134b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:44:16.480 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:16.495+0000 7f90d134b640 1 -- 172.21.15.88:0/2536790102 >> v1:172.21.15.88:6789/0 conn(0x7f90cc0b1f60 legacy=0x7f90cc143600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:16.480 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:16.496+0000 7f90d134b640 1 -- 172.21.15.88:0/2536790102 shutdown_connections 2024-03-29T10:44:16.480 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:16.496+0000 7f90d134b640 1 -- 172.21.15.88:0/2536790102 >> 172.21.15.88:0/2536790102 conn(0x7f90cc066a00 msgr2=0x7f90cc066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:44:16.480 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:16.497+0000 7f90d134b640 1 -- 172.21.15.88:0/2536790102 shutdown_connections 2024-03-29T10:44:16.482 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:16.497+0000 7f90d134b640 1 -- 172.21.15.88:0/2536790102 wait complete. 2024-03-29T10:44:16.482 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:16.498+0000 7f90d134b640 10 monclient: build_initial_monmap 2024-03-29T10:44:16.483 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:16.498+0000 7f90d134b640 10 monclient: monmap: 2024-03-29T10:44:16.483 INFO:teuthology.misc.health.smithi088.stderr:epoch 0 2024-03-29T10:44:16.483 INFO:teuthology.misc.health.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:44:16.483 INFO:teuthology.misc.health.smithi088.stderr:last_changed 0.000000 2024-03-29T10:44:16.483 INFO:teuthology.misc.health.smithi088.stderr:created 0.000000 2024-03-29T10:44:16.483 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 0 (unknown) 2024-03-29T10:44:16.483 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:44:16.483 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-03-29T10:44:16.483 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-29T10:44:16.483 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-c 2024-03-29T10:44:16.483 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:44:16.483 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:16.499+0000 7f90d134b640 1 librados: starting msgr at 2024-03-29T10:44:16.483 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:16.499+0000 7f90d134b640 1 librados: starting objecter 2024-03-29T10:44:16.483 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:16.499+0000 7f90d134b640 1 Processor -- start 2024-03-29T10:44:16.484 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:16.500+0000 7f90d134b640 1 -- start start 2024-03-29T10:44:16.484 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:16.500+0000 7f90d134b640 1 librados: setting wanted keys 2024-03-29T10:44:16.484 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:16.500+0000 7f90d134b640 1 librados: calling monclient init 2024-03-29T10:44:16.484 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:16.500+0000 7f90d134b640 10 monclient: init 2024-03-29T10:44:16.484 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:16.501+0000 7f90d134b640 10 monclient: _reopen_session rank -1 2024-03-29T10:44:16.484 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:16.501+0000 7f90d134b640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:44:16.485 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:16.501+0000 7f90d134b640 10 monclient(hunting): picked mon.noname-a con 0x7f90cc0b1f60 addr v1:172.21.15.88:6789/0 2024-03-29T10:44:16.485 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:16.501+0000 7f90d134b640 10 monclient(hunting): picked mon.noname-b con 0x7f90cc143d10 addr v1:172.21.15.196:6789/0 2024-03-29T10:44:16.485 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:16.501+0000 7f90d134b640 10 monclient(hunting): picked mon.noname-c con 0x7f90cc06fac0 addr v1:172.21.15.88:6790/0 2024-03-29T10:44:16.485 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:16.501+0000 7f90d134b640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90cc0ab420 con 0x7f90cc0b1f60 2024-03-29T10:44:16.485 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:16.501+0000 7f90d134b640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90cc0b1300 con 0x7f90cc143d10 2024-03-29T10:44:16.485 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:16.501+0000 7f90d134b640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90cc0ac5d0 con 0x7f90cc06fac0 2024-03-29T10:44:16.485 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:16.501+0000 7f90d134b640 10 monclient(hunting): _renew_subs 2024-03-29T10:44:16.485 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:16.501+0000 7f90d134b640 10 monclient(hunting): authenticate will time out at 978.737185s 2024-03-29T10:44:16.485 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:16.501+0000 7f90caffd640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f90cc0b1f60 0x7f90cc0aad10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:57936/0 (socket says 172.21.15.88:57936) 2024-03-29T10:44:16.485 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:16.501+0000 7f90caffd640 1 -- 172.21.15.88:0/449688666 learned_addr learned my addr 172.21.15.88:0/449688666 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-29T10:44:16.485 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:16.501+0000 7f90c37fe640 10 client.?.objecter ms_handle_connect 0x7f90cc0b1f60 2024-03-29T10:44:16.485 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:16.501+0000 7f90c37fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:44:16.486 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:16.502+0000 7f90c37fe640 1 -- 172.21.15.88:0/449688666 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2453759851 0 0) 0x7f90cc155d70 con 0x7f90cc0b1f60 2024-03-29T10:44:16.486 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:16.502+0000 7f90c37fe640 10 monclient(hunting): _negotiate 2024-03-29T10:44:16.486 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:16.502+0000 7f90c37fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:16.486 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:16.502+0000 7f90c37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:16.486 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:16.502+0000 7f90c37fe640 10 monclient(hunting): my global_id is 4347 2024-03-29T10:44:16.486 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:16.502+0000 7f90c37fe640 1 -- 172.21.15.88:0/449688666 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f90ac004eb0 con 0x7f90cc0b1f60 2024-03-29T10:44:16.487 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:16.502+0000 7f90c37fe640 1 -- 172.21.15.88:0/449688666 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2672014689 0 0) 0x7f90ac004eb0 con 0x7f90cc0b1f60 2024-03-29T10:44:16.487 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:16.502+0000 7f90c37fe640 1 -- 172.21.15.88:0/449688666 --> v1:172.21.15.88:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f90cc155d70 con 0x7f90cc0b1f60 2024-03-29T10:44:16.487 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:16.502+0000 7f90c37fe640 1 -- 172.21.15.88:0/449688666 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f90bc0015b0 con 0x7f90cc0b1f60 2024-03-29T10:44:16.487 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:16.502+0000 7f90c37fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:16.487 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:16.502+0000 7f90c37fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:44:16.487 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:16.502+0000 7f90c37fe640 10 monclient(hunting): dump: 2024-03-29T10:44:16.487 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:44:16.487 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:44:16.487 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:44:16.487 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:44:16.487 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:16.487 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:44:16.487 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:44:16.487 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:44:16.487 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:44:16.487 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:44:16.488 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:16.502+0000 7f90c37fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:44:16.488 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:16.502+0000 7f90c37fe640 1 -- 172.21.15.88:0/449688666 <== mon.0 v1:172.21.15.88:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2768078936 0 0) 0x7f90cc155d70 con 0x7f90cc0b1f60 2024-03-29T10:44:16.488 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:16.502+0000 7f90c37fe640 1 -- 172.21.15.88:0/449688666 >> v1:172.21.15.88:6790/0 conn(0x7f90cc06fac0 legacy=0x7f90cc13fce0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:44:16.490 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:16.505+0000 7f90c37fe640 1 -- 172.21.15.88:0/449688666 >> v1:172.21.15.196:6789/0 conn(0x7f90cc143d10 legacy=0x7f90cc1e4440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:16.490 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:16.505+0000 7f90c37fe640 10 monclient: _finish_hunting 0 2024-03-29T10:44:16.490 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:16.505+0000 7f90c37fe640 1 monclient: found mon.a 2024-03-29T10:44:16.490 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:16.505+0000 7f90c37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:44:16.490 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:16.505+0000 7f90c37fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-03-29T10:44:16.490 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:16.505+0000 7f90c37fe640 1 -- 172.21.15.88:0/449688666 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90cc1e3f20 con 0x7f90cc0b1f60 2024-03-29T10:44:16.490 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:16.505+0000 7f90c37fe640 10 monclient: _finish_auth 0 2024-03-29T10:44:16.490 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:16.505+0000 7f90c37fe640 10 monclient: _check_auth_tickets 2024-03-29T10:44:16.490 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:16.505+0000 7f90c37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:44:16.491 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:16.505+0000 7f90c37fe640 10 client.?.objecter ms_handle_connect 0x7f90cc143d10 2024-03-29T10:44:16.491 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:16.505+0000 7f90c37fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:44:16.491 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:16.505+0000 7f90c37fe640 1 -- 172.21.15.88:0/449688666 <== mon.0 v1:172.21.15.88:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f90bc001710 con 0x7f90cc0b1f60 2024-03-29T10:44:16.491 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:16.505+0000 7f90c37fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:44:16.491 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:16.505+0000 7f90c37fe640 1 -- 172.21.15.88:0/449688666 <== mon.0 v1:172.21.15.88:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f90bc004d90 con 0x7f90cc0b1f60 2024-03-29T10:44:16.491 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:16.505+0000 7f90c37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:16.491 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:16.505+0000 7f90c37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:44:16.491 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:16.505+0000 7f90c37fe640 10 monclient: dump: 2024-03-29T10:44:16.491 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:44:16.491 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:44:16.491 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:44:16.491 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:44:16.491 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:16.491 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:44:16.492 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:44:16.492 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:44:16.492 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:44:16.492 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:44:16.492 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:16.505+0000 7f90c97fa640 4 set_mon_vals no callback set 2024-03-29T10:44:16.492 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:16.505+0000 7f90d134b640 5 monclient: authenticate success, global_id 4347 2024-03-29T10:44:16.492 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:16.505+0000 7f90d134b640 10 monclient: _renew_subs 2024-03-29T10:44:16.492 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:16.505+0000 7f90d134b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-03-29T10:44:16.492 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:16.505+0000 7f90d134b640 1 -- 172.21.15.88:0/449688666 --> v1:172.21.15.88:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90cc0a9f00 con 0x7f90cc0b1f60 2024-03-29T10:44:16.492 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:16.505+0000 7f90d134b640 10 client.4347.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:44:16.492 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:16.505+0000 7f90d134b640 10 monclient: _renew_subs 2024-03-29T10:44:16.492 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:16.505+0000 7f90d134b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-03-29T10:44:16.492 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:16.505+0000 7f90d134b640 1 -- 172.21.15.88:0/449688666 --> v1:172.21.15.88:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f90cc1403f0 con 0x7f90cc0b1f60 2024-03-29T10:44:16.492 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:16.505+0000 7f90d134b640 1 librados: init done 2024-03-29T10:44:16.492 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:16.507+0000 7f90d134b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:44:16.493 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:16.507+0000 7f90d134b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:44:16.493 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:16.507+0000 7f90d134b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-03-29T10:44:16.493 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:16.507+0000 7f90d134b640 1 -- 172.21.15.88:0/449688666 --> v1:172.21.15.88:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90cc1e3d80 con 0x7f90cc0b1f60 2024-03-29T10:44:16.493 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:16.510+0000 7f90c37fe640 1 -- 172.21.15.88:0/449688666 <== mon.0 v1:172.21.15.88:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4235629220 0 0) 0x7f90bc01ade0 con 0x7f90cc0b1f60 2024-03-29T10:44:16.493 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:16.511+0000 7f90c3fff640 10 monclient: get_auth_request con 0x7f90ac06db50 auth_method 0 2024-03-29T10:44:16.494 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:16.511+0000 7f90c37fe640 1 -- 172.21.15.88:0/449688666 <== mon.0 v1:172.21.15.88:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4257673582 0 0) 0x7f90bc084c60 con 0x7f90cc0b1f60 2024-03-29T10:44:16.494 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:16.511+0000 7f90c37fe640 10 client.4347.objecter ms_dispatch 0x7f90cc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:44:16.494 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:16.511+0000 7f90c37fe640 3 client.4347.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:44:16.494 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:16.511+0000 7f90c37fe640 3 client.4347.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:44:16.494 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:16.511+0000 7f90c37fe640 20 client.4347.objecter dump_active .. 0 homeless 2024-03-29T10:44:16.494 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:16.511+0000 7f90c37fe640 10 client.4347.objecter ms_handle_connect 0x7f90ac06db50 2024-03-29T10:44:16.501 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:16.518+0000 7f90c37fe640 1 -- 172.21.15.88:0/449688666 <== mon.0 v1:172.21.15.88:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f90cc1e3d80 con 0x7f90cc0b1f60 2024-03-29T10:44:16.501 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:16.518+0000 7f90c37fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:44:16.501 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:16.518+0000 7f90c37fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:44:16.829 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:16.846+0000 7f90d134b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:44:16.830 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:16.846+0000 7f90d134b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:44:16.830 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:16.846+0000 7f90d134b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-03-29T10:44:16.830 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:16.846+0000 7f90d134b640 1 -- 172.21.15.88:0/449688666 --> v1:172.21.15.88:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f90cc1e3d80 con 0x7f90cc0b1f60 2024-03-29T10:44:16.832 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:16.849+0000 7f90c37fe640 1 -- 172.21.15.88:0/449688666 <== mon.0 v1:172.21.15.88:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 344506374) 0x7f90cc1e3d80 con 0x7f90cc0b1f60 2024-03-29T10:44:16.832 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:16.849+0000 7f90c37fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:44:16.832 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:16.849+0000 7f90c37fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:44:16.833 INFO:teuthology.misc.health.smithi088.stdout: 2024-03-29T10:44:16.833 INFO:teuthology.misc.health.smithi088.stdout:{"epoch":10,"fsid":"869f5188-f92b-42dd-a46d-8c646e0388a9","created":"2024-03-29T10:40:40.285087+0000","modified":"2024-03-29T10:40:42.536230+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:40:42.536230+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a26c0c1c-8de0-438c-a3f6-3df10ea61574","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5de24915-8dec-49b6-8b16-f3170197c591","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ac054af3-d589-4565-9644-e6af75975c53","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3ec4f045-c714-4a22-bd1a-e07330b45ab3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"cddb9867-8e30-4951-9b13-3d1e78b9c45a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dc6d9911-785d-432f-8b53-56a1ccab302f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"77c88437-b587-496e-8b4d-1824ea8b5305","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ce963d96-5a8e-44c3-99de-c0dbebe234ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:44:16.838 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:16.853+0000 7f90d134b640 10 librados: watch_flush enter 2024-03-29T10:44:16.838 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:16.853+0000 7f90d134b640 10 librados: watch_flush exit 2024-03-29T10:44:16.838 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:16.853+0000 7f90d134b640 20 client.4347.objecter shutdown clearing up homeless session... 2024-03-29T10:44:16.838 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:16.853+0000 7f90d134b640 10 client.4347.objecter successfully canceled tick 2024-03-29T10:44:16.838 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:16.853+0000 7f90d134b640 1 -- 172.21.15.88:0/449688666 >> v1:172.21.15.88:6816/1752861349 conn(0x7f90ac06db50 legacy=0x7f90ac070010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:16.839 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:16.853+0000 7f90d134b640 10 monclient: shutdown 2024-03-29T10:44:16.839 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:16.853+0000 7f90d134b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:44:16.839 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:16.853+0000 7f90d134b640 1 -- 172.21.15.88:0/449688666 >> v1:172.21.15.88:6789/0 conn(0x7f90cc0b1f60 legacy=0x7f90cc0aad10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:16.839 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:16.853+0000 7f90d134b640 1 -- 172.21.15.88:0/449688666 shutdown_connections 2024-03-29T10:44:16.839 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:16.853+0000 7f90d134b640 1 -- 172.21.15.88:0/449688666 >> 172.21.15.88:0/449688666 conn(0x7f90cc066a00 msgr2=0x7f90cc146a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:44:16.839 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:16.856+0000 7f90d134b640 1 -- 172.21.15.88:0/449688666 shutdown_connections 2024-03-29T10:44:16.839 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:16.857+0000 7f90d134b640 1 -- 172.21.15.88:0/449688666 wait complete. 2024-03-29T10:44:16.841 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:16.858+0000 7f90d134b640 1 librados: shutdown 2024-03-29T10:44:16.859 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:44:23.666 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:44:23.812 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:23.829+0000 7f89d3433640 10 monclient: get_monmap_and_config 2024-03-29T10:44:23.813 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:23.829+0000 7f89d3433640 10 monclient: build_initial_monmap 2024-03-29T10:44:23.813 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:23.829+0000 7f89d3433640 10 monclient: monmap: 2024-03-29T10:44:23.813 INFO:teuthology.misc.health.smithi088.stderr:epoch 0 2024-03-29T10:44:23.813 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:44:23.813 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:44:23.830825+0000 2024-03-29T10:44:23.813 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:44:23.830825+0000 2024-03-29T10:44:23.813 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 0 (unknown) 2024-03-29T10:44:23.813 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:44:23.813 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-03-29T10:44:23.813 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-29T10:44:23.813 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-b 2024-03-29T10:44:23.813 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:44:23.814 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:23.830+0000 7f89d3433640 1 Processor -- start 2024-03-29T10:44:23.815 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:23.830+0000 7f89d3433640 1 -- start start 2024-03-29T10:44:23.815 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:23.830+0000 7f89d3433640 10 monclient: init 2024-03-29T10:44:23.815 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:23.831+0000 7f89d3433640 10 monclient: _reopen_session rank -1 2024-03-29T10:44:23.815 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:23.831+0000 7f89d3433640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:44:23.815 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:23.831+0000 7f89d3433640 10 monclient(hunting): picked mon.noname-a con 0x7f89cc1457f0 addr v1:172.21.15.88:6789/0 2024-03-29T10:44:23.815 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:23.831+0000 7f89d3433640 10 monclient(hunting): picked mon.noname-c con 0x7f89cc13f650 addr v1:172.21.15.196:6789/0 2024-03-29T10:44:23.815 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:23.831+0000 7f89d3433640 10 monclient(hunting): picked mon.noname-b con 0x7f89cc0a9d70 addr v1:172.21.15.88:6790/0 2024-03-29T10:44:23.815 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:23.831+0000 7f89d3433640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89cc0aaef0 con 0x7f89cc1457f0 2024-03-29T10:44:23.815 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:23.831+0000 7f89d3433640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89cc145bd0 con 0x7f89cc13f650 2024-03-29T10:44:23.815 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:23.831+0000 7f89d3433640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89cc145eb0 con 0x7f89cc0a9d70 2024-03-29T10:44:23.816 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:23.831+0000 7f89d3433640 10 monclient(hunting): _renew_subs 2024-03-29T10:44:23.816 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:23.831+0000 7f89d3433640 10 monclient(hunting): authenticate will time out at 986.067765s 2024-03-29T10:44:23.816 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:23.833+0000 7f89d09ce640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f89cc13f650 0x7f89cc0b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.88:50988/0 (socket says 172.21.15.88:50988) 2024-03-29T10:44:23.816 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:23.833+0000 7f89d09ce640 1 -- 172.21.15.88:0/2229466975 learned_addr learned my addr 172.21.15.88:0/2229466975 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-29T10:44:23.819 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:23.835+0000 7f89c3fff640 1 -- 172.21.15.88:0/2229466975 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1432128185 0 0) 0x7f89cc145bd0 con 0x7f89cc13f650 2024-03-29T10:44:23.819 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:23.835+0000 7f89c3fff640 10 monclient(hunting): _negotiate 2024-03-29T10:44:23.819 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:23.835+0000 7f89c3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:23.819 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:23.835+0000 7f89c3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:23.819 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:23.835+0000 7f89c3fff640 10 monclient(hunting): my global_id is 4345 2024-03-29T10:44:23.822 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:23.835+0000 7f89c3fff640 1 -- 172.21.15.88:0/2229466975 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f89b8003520 con 0x7f89cc13f650 2024-03-29T10:44:23.822 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:23.835+0000 7f89c3fff640 1 -- 172.21.15.88:0/2229466975 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3648877376 0 0) 0x7f89cc0aaef0 con 0x7f89cc1457f0 2024-03-29T10:44:23.822 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:23.835+0000 7f89c3fff640 10 monclient(hunting): _negotiate 2024-03-29T10:44:23.822 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:23.835+0000 7f89c3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:23.822 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:23.835+0000 7f89c3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:23.822 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:23.835+0000 7f89c3fff640 10 monclient(hunting): my global_id is 4350 2024-03-29T10:44:23.822 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:23.835+0000 7f89c3fff640 1 -- 172.21.15.88:0/2229466975 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f89cc145bd0 con 0x7f89cc1457f0 2024-03-29T10:44:23.822 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:23.836+0000 7f89c3fff640 1 -- 172.21.15.88:0/2229466975 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3052908554 0 0) 0x7f89cc145bd0 con 0x7f89cc1457f0 2024-03-29T10:44:23.822 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:23.836+0000 7f89c3fff640 1 -- 172.21.15.88:0/2229466975 >> v1:172.21.15.88:6790/0 conn(0x7f89cc0a9d70 legacy=0x7f89cc14d8c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:44:23.822 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:23.836+0000 7f89c3fff640 1 -- 172.21.15.88:0/2229466975 >> v1:172.21.15.196:6789/0 conn(0x7f89cc13f650 legacy=0x7f89cc0b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:23.822 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:23.836+0000 7f89c3fff640 10 monclient: _finish_hunting 0 2024-03-29T10:44:23.822 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:23.836+0000 7f89c3fff640 1 monclient: found mon.noname-a 2024-03-29T10:44:23.823 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:23.836+0000 7f89c3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:44:23.823 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:23.836+0000 7f89c3fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.88:6789/0 2024-03-29T10:44:23.823 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:23.836+0000 7f89c3fff640 1 -- 172.21.15.88:0/2229466975 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89cc146340 con 0x7f89cc1457f0 2024-03-29T10:44:23.823 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:23.836+0000 7f89c3fff640 10 monclient: _finish_auth 0 2024-03-29T10:44:23.823 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:23.836+0000 7f89c3fff640 10 monclient: _check_auth_tickets 2024-03-29T10:44:23.823 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:23.836+0000 7f89c3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:44:23.823 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:23.836+0000 7f89c3fff640 1 -- 172.21.15.88:0/2229466975 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f89c8001520 con 0x7f89cc1457f0 2024-03-29T10:44:23.823 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:23.836+0000 7f89c3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:23.823 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:23.836+0000 7f89c3fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:44:23.823 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:23.836+0000 7f89c3fff640 10 monclient: dump: 2024-03-29T10:44:23.823 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:44:23.823 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:44:23.823 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:44:23.824 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:44:23.824 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:23.824 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:44:23.824 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:44:23.824 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:44:23.824 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:44:23.824 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:44:23.824 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:23.836+0000 7f89d3433640 5 monclient: authenticate success, global_id 4350 2024-03-29T10:44:23.824 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:23.836+0000 7f89d3433640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:44:23.824 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:23.836+0000 7f89c3fff640 1 -- 172.21.15.88:0/2229466975 <== mon.0 v1:172.21.15.88:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f89c8001680 con 0x7f89cc1457f0 2024-03-29T10:44:23.825 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:23.836+0000 7f89c3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:44:23.825 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:23.836+0000 7f89c3fff640 1 -- 172.21.15.88:0/2229466975 <== mon.0 v1:172.21.15.88:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f89c8001a90 con 0x7f89cc1457f0 2024-03-29T10:44:23.825 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:23.836+0000 7f89c3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:23.825 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:23.836+0000 7f89c3fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:44:23.825 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:23.836+0000 7f89c3fff640 10 monclient: dump: 2024-03-29T10:44:23.825 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:44:23.825 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:44:23.825 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:44:23.825 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:44:23.825 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:23.825 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:44:23.825 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:44:23.825 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:44:23.826 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:44:23.826 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:44:23.826 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:23.836+0000 7f89d3433640 10 monclient: get_monmap_and_config success 2024-03-29T10:44:23.826 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:23.836+0000 7f89d3433640 4 set_mon_vals no callback set 2024-03-29T10:44:23.826 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:23.837+0000 7f89d3433640 10 monclient: shutdown 2024-03-29T10:44:23.826 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:23.837+0000 7f89d3433640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:44:23.826 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:23.837+0000 7f89d3433640 1 -- 172.21.15.88:0/2229466975 >> v1:172.21.15.88:6789/0 conn(0x7f89cc1457f0 legacy=0x7f89cc13ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:23.826 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:23.837+0000 7f89d3433640 1 -- 172.21.15.88:0/2229466975 shutdown_connections 2024-03-29T10:44:23.826 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:23.837+0000 7f89d3433640 1 -- 172.21.15.88:0/2229466975 >> 172.21.15.88:0/2229466975 conn(0x7f89cc066a00 msgr2=0x7f89cc066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:44:23.826 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:23.837+0000 7f89d3433640 1 -- 172.21.15.88:0/2229466975 shutdown_connections 2024-03-29T10:44:23.826 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:23.837+0000 7f89d3433640 1 -- 172.21.15.88:0/2229466975 wait complete. 2024-03-29T10:44:23.827 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:23.838+0000 7f89d3433640 10 monclient: build_initial_monmap 2024-03-29T10:44:23.827 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:23.838+0000 7f89d3433640 10 monclient: monmap: 2024-03-29T10:44:23.827 INFO:teuthology.misc.health.smithi088.stderr:epoch 0 2024-03-29T10:44:23.827 INFO:teuthology.misc.health.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:44:23.827 INFO:teuthology.misc.health.smithi088.stderr:last_changed 0.000000 2024-03-29T10:44:23.827 INFO:teuthology.misc.health.smithi088.stderr:created 0.000000 2024-03-29T10:44:23.827 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 0 (unknown) 2024-03-29T10:44:23.827 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:44:23.827 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-03-29T10:44:23.827 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-29T10:44:23.827 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-c 2024-03-29T10:44:23.828 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:44:23.828 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:23.839+0000 7f89d3433640 1 librados: starting msgr at 2024-03-29T10:44:23.828 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:23.839+0000 7f89d3433640 1 librados: starting objecter 2024-03-29T10:44:23.830 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:23.839+0000 7f89d3433640 1 Processor -- start 2024-03-29T10:44:23.830 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:23.839+0000 7f89d3433640 1 -- start start 2024-03-29T10:44:23.830 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:23.839+0000 7f89d3433640 1 librados: setting wanted keys 2024-03-29T10:44:23.831 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:23.839+0000 7f89d3433640 1 librados: calling monclient init 2024-03-29T10:44:23.831 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:23.839+0000 7f89d3433640 10 monclient: init 2024-03-29T10:44:23.831 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:23.840+0000 7f89d3433640 10 monclient: _reopen_session rank -1 2024-03-29T10:44:23.831 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:23.840+0000 7f89d3433640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:44:23.831 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:23.842+0000 7f89d3433640 10 monclient(hunting): picked mon.noname-a con 0x7f89cc0a9d70 addr v1:172.21.15.88:6789/0 2024-03-29T10:44:23.831 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:23.842+0000 7f89d3433640 10 monclient(hunting): picked mon.noname-b con 0x7f89cc13f650 addr v1:172.21.15.196:6789/0 2024-03-29T10:44:23.831 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:23.843+0000 7f89d3433640 10 monclient(hunting): picked mon.noname-c con 0x7f89cc1457f0 addr v1:172.21.15.88:6790/0 2024-03-29T10:44:23.831 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:23.843+0000 7f89d3433640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89cc141690 con 0x7f89cc0a9d70 2024-03-29T10:44:23.831 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:23.843+0000 7f89d09ce640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f89cc13f650 0x7f89cc1e4170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.88:51422/0 (socket says 172.21.15.88:51422) 2024-03-29T10:44:23.831 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:23.843+0000 7f89d3433640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89cc141820 con 0x7f89cc13f650 2024-03-29T10:44:23.832 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:23.843+0000 7f89d09ce640 1 -- 172.21.15.88:0/1457763085 learned_addr learned my addr 172.21.15.88:0/1457763085 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-29T10:44:23.832 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:23.843+0000 7f89d3433640 1 -- 172.21.15.88:0/1457763085 --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89cc1419b0 con 0x7f89cc1457f0 2024-03-29T10:44:23.832 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:23.843+0000 7f89d3433640 10 monclient(hunting): _renew_subs 2024-03-29T10:44:23.832 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:23.843+0000 7f89d3433640 10 monclient(hunting): authenticate will time out at 986.079619s 2024-03-29T10:44:23.832 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:23.846+0000 7f89c1ffb640 10 client.?.objecter ms_handle_connect 0x7f89cc0a9d70 2024-03-29T10:44:23.832 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:23.846+0000 7f89c1ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:44:23.832 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:23.846+0000 7f89c1ffb640 1 -- 172.21.15.88:0/1457763085 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 907355006 0 0) 0x7f89cc141690 con 0x7f89cc0a9d70 2024-03-29T10:44:23.832 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:23.846+0000 7f89c1ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:44:23.833 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:23.846+0000 7f89c1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:23.833 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:23.846+0000 7f89c1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:23.833 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:23.846+0000 7f89c1ffb640 10 monclient(hunting): my global_id is 4353 2024-03-29T10:44:23.833 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:23.846+0000 7f89c1ffb640 1 -- 172.21.15.88:0/1457763085 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f89b0004eb0 con 0x7f89cc0a9d70 2024-03-29T10:44:23.833 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:23.846+0000 7f89c1ffb640 10 client.?.objecter ms_handle_connect 0x7f89cc13f650 2024-03-29T10:44:23.833 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:23.846+0000 7f89c1ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:44:23.833 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:23.846+0000 7f89c1ffb640 1 -- 172.21.15.88:0/1457763085 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2063738371 0 0) 0x7f89cc141820 con 0x7f89cc13f650 2024-03-29T10:44:23.833 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:23.846+0000 7f89c1ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:44:23.833 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:23.846+0000 7f89c1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:23.833 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:23.846+0000 7f89c1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:23.834 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:23.846+0000 7f89c1ffb640 10 monclient(hunting): my global_id is 4348 2024-03-29T10:44:23.834 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:23.846+0000 7f89c1ffb640 1 -- 172.21.15.88:0/1457763085 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f89cc141690 con 0x7f89cc13f650 2024-03-29T10:44:23.834 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:23.846+0000 7f89c1ffb640 10 client.?.objecter ms_handle_connect 0x7f89cc1457f0 2024-03-29T10:44:23.834 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:23.846+0000 7f89c1ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:44:23.834 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:23.846+0000 7f89c1ffb640 1 -- 172.21.15.88:0/1457763085 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3357751465 0 0) 0x7f89b0004eb0 con 0x7f89cc0a9d70 2024-03-29T10:44:23.834 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:23.846+0000 7f89c1ffb640 1 -- 172.21.15.88:0/1457763085 --> v1:172.21.15.88:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f89cc141820 con 0x7f89cc0a9d70 2024-03-29T10:44:23.834 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:23.846+0000 7f89c1ffb640 1 -- 172.21.15.88:0/1457763085 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f89c8001ad0 con 0x7f89cc0a9d70 2024-03-29T10:44:23.834 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:23.846+0000 7f89c1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:23.834 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:23.846+0000 7f89c1ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:44:23.834 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:23.846+0000 7f89c1ffb640 10 monclient(hunting): dump: 2024-03-29T10:44:23.835 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:44:23.835 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:44:23.835 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:44:23.835 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:44:23.835 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:23.835 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:44:23.835 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:44:23.835 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:44:23.835 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:44:23.835 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:44:23.835 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:23.846+0000 7f89c1ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:44:23.835 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:23.847+0000 7f89c1ffb640 1 -- 172.21.15.88:0/1457763085 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 937328594 0 0) 0x7f89cc1419b0 con 0x7f89cc1457f0 2024-03-29T10:44:23.836 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:23.847+0000 7f89c1ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:44:23.836 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:23.847+0000 7f89c1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:23.836 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:23.847+0000 7f89c1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:23.836 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:23.847+0000 7f89c1ffb640 10 monclient(hunting): my global_id is 4376 2024-03-29T10:44:23.836 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:23.847+0000 7f89c1ffb640 1 -- 172.21.15.88:0/1457763085 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f89b0004eb0 con 0x7f89cc1457f0 2024-03-29T10:44:23.836 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:23.847+0000 7f89c1ffb640 1 -- 172.21.15.88:0/1457763085 <== mon.0 v1:172.21.15.88:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 791908540 0 0) 0x7f89cc141820 con 0x7f89cc0a9d70 2024-03-29T10:44:23.836 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:23.847+0000 7f89c1ffb640 1 -- 172.21.15.88:0/1457763085 >> v1:172.21.15.88:6790/0 conn(0x7f89cc1457f0 legacy=0x7f89cc1e4880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:23.836 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:23.847+0000 7f89c1ffb640 1 -- 172.21.15.88:0/1457763085 >> v1:172.21.15.196:6789/0 conn(0x7f89cc13f650 legacy=0x7f89cc1e4170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:23.836 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:23.847+0000 7f89c1ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:44:23.836 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:23.847+0000 7f89c1ffb640 1 monclient: found mon.a 2024-03-29T10:44:23.836 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:23.847+0000 7f89c1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:44:23.836 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:23.847+0000 7f89c1ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-03-29T10:44:23.837 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:23.847+0000 7f89c1ffb640 1 -- 172.21.15.88:0/1457763085 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89cc1d9820 con 0x7f89cc0a9d70 2024-03-29T10:44:23.837 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:23.847+0000 7f89c1ffb640 10 monclient: _finish_auth 0 2024-03-29T10:44:23.837 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:23.847+0000 7f89c1ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:44:23.837 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:23.847+0000 7f89c1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:44:23.837 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:23.847+0000 7f89d3433640 5 monclient: authenticate success, global_id 4353 2024-03-29T10:44:23.837 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:23.847+0000 7f89d3433640 10 monclient: _renew_subs 2024-03-29T10:44:23.837 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:23.847+0000 7f89d3433640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-03-29T10:44:23.837 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:23.847+0000 7f89d3433640 1 -- 172.21.15.88:0/1457763085 --> v1:172.21.15.88:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89cc1d9af0 con 0x7f89cc0a9d70 2024-03-29T10:44:23.840 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:23.847+0000 7f89d3433640 10 client.4353.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:44:23.840 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:23.847+0000 7f89d3433640 10 monclient: _renew_subs 2024-03-29T10:44:23.840 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:23.847+0000 7f89d3433640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-03-29T10:44:23.840 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:23.847+0000 7f89d3433640 1 -- 172.21.15.88:0/1457763085 --> v1:172.21.15.88:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f89cc1d7030 con 0x7f89cc0a9d70 2024-03-29T10:44:23.840 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:23.847+0000 7f89d3433640 1 librados: init done 2024-03-29T10:44:23.840 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:23.850+0000 7f89b77fe640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:44:23.841 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:23.850+0000 7f89b77fe640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:44:23.841 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:23.850+0000 7f89b77fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-03-29T10:44:23.841 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:23.850+0000 7f89b77fe640 1 -- 172.21.15.88:0/1457763085 --> v1:172.21.15.88:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89cc0b1d60 con 0x7f89cc0a9d70 2024-03-29T10:44:23.841 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:23.851+0000 7f89c1ffb640 1 -- 172.21.15.88:0/1457763085 <== mon.0 v1:172.21.15.88:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f89c8001d10 con 0x7f89cc0a9d70 2024-03-29T10:44:23.841 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:23.851+0000 7f89c1ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:44:23.841 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:23.851+0000 7f89c37fe640 4 set_mon_vals no callback set 2024-03-29T10:44:23.841 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:23.852+0000 7f89c1ffb640 1 -- 172.21.15.88:0/1457763085 <== mon.0 v1:172.21.15.88:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f89c80010e0 con 0x7f89cc0a9d70 2024-03-29T10:44:23.841 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:23.852+0000 7f89c1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:23.841 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:23.852+0000 7f89c1ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:44:23.841 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:23.852+0000 7f89c1ffb640 10 monclient: dump: 2024-03-29T10:44:23.841 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:44:23.841 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:44:23.841 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:44:23.841 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:44:23.842 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:23.842 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:44:23.842 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:44:23.842 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:44:23.842 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:44:23.842 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:44:23.842 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:23.852+0000 7f89c1ffb640 1 -- 172.21.15.88:0/1457763085 <== mon.0 v1:172.21.15.88:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4235629220 0 0) 0x7f89c8019930 con 0x7f89cc0a9d70 2024-03-29T10:44:23.842 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:23.853+0000 7f89d09ce640 10 monclient: get_auth_request con 0x7f89b006de80 auth_method 0 2024-03-29T10:44:23.842 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:23.853+0000 7f89c1ffb640 1 -- 172.21.15.88:0/1457763085 <== mon.0 v1:172.21.15.88:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4257673582 0 0) 0x7f89c8083910 con 0x7f89cc0a9d70 2024-03-29T10:44:23.842 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:23.853+0000 7f89c1ffb640 10 client.4353.objecter ms_dispatch 0x7f89cc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:44:23.842 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:23.853+0000 7f89c1ffb640 3 client.4353.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:44:23.842 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:23.853+0000 7f89c1ffb640 3 client.4353.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:44:23.842 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:23.853+0000 7f89c1ffb640 20 client.4353.objecter dump_active .. 0 homeless 2024-03-29T10:44:23.842 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:23.853+0000 7f89c1ffb640 10 client.4353.objecter ms_handle_connect 0x7f89b006de80 2024-03-29T10:44:23.843 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:23.860+0000 7f89c1ffb640 1 -- 172.21.15.88:0/1457763085 <== mon.0 v1:172.21.15.88:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f89c804f0a0 con 0x7f89cc0a9d70 2024-03-29T10:44:23.843 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:23.860+0000 7f89c1ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:44:23.843 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:23.860+0000 7f89c1ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:44:24.166 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:24.183+0000 7f89b77fe640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:44:24.166 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:24.183+0000 7f89b77fe640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:44:24.166 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:24.183+0000 7f89b77fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-03-29T10:44:24.166 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:24.183+0000 7f89b77fe640 1 -- 172.21.15.88:0/1457763085 --> v1:172.21.15.88:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f89cc075f10 con 0x7f89cc0a9d70 2024-03-29T10:44:24.167 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:24.184+0000 7f89c1ffb640 1 -- 172.21.15.88:0/1457763085 <== mon.0 v1:172.21.15.88:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 344506374) 0x7f89c80511a0 con 0x7f89cc0a9d70 2024-03-29T10:44:24.167 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:24.184+0000 7f89c1ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:44:24.167 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:24.184+0000 7f89c1ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:44:24.167 INFO:teuthology.misc.health.smithi088.stdout: 2024-03-29T10:44:24.168 INFO:teuthology.misc.health.smithi088.stdout:{"epoch":10,"fsid":"869f5188-f92b-42dd-a46d-8c646e0388a9","created":"2024-03-29T10:40:40.285087+0000","modified":"2024-03-29T10:40:42.536230+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:40:42.536230+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a26c0c1c-8de0-438c-a3f6-3df10ea61574","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5de24915-8dec-49b6-8b16-f3170197c591","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ac054af3-d589-4565-9644-e6af75975c53","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3ec4f045-c714-4a22-bd1a-e07330b45ab3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"cddb9867-8e30-4951-9b13-3d1e78b9c45a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dc6d9911-785d-432f-8b53-56a1ccab302f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"77c88437-b587-496e-8b4d-1824ea8b5305","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ce963d96-5a8e-44c3-99de-c0dbebe234ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:44:24.170 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:24.187+0000 7f89b77fe640 10 librados: watch_flush enter 2024-03-29T10:44:24.171 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:24.188+0000 7f89b77fe640 10 librados: watch_flush exit 2024-03-29T10:44:24.171 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:24.188+0000 7f89b77fe640 20 client.4353.objecter shutdown clearing up homeless session... 2024-03-29T10:44:24.171 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:24.188+0000 7f89b77fe640 10 client.4353.objecter successfully canceled tick 2024-03-29T10:44:24.171 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:24.188+0000 7f89b77fe640 1 -- 172.21.15.88:0/1457763085 >> v1:172.21.15.88:6816/1752861349 conn(0x7f89b006de80 legacy=0x7f89b0070340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:24.171 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:24.188+0000 7f89b77fe640 10 monclient: shutdown 2024-03-29T10:44:24.171 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:24.188+0000 7f89b77fe640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:44:24.171 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:24.188+0000 7f89b77fe640 1 -- 172.21.15.88:0/1457763085 >> v1:172.21.15.88:6789/0 conn(0x7f89cc0a9d70 legacy=0x7f89cc1d6740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:24.172 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:24.189+0000 7f89b77fe640 1 -- 172.21.15.88:0/1457763085 shutdown_connections 2024-03-29T10:44:24.172 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:24.189+0000 7f89b77fe640 1 -- 172.21.15.88:0/1457763085 >> 172.21.15.88:0/1457763085 conn(0x7f89cc066a00 msgr2=0x7f89cc0b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:44:24.172 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:24.189+0000 7f89b77fe640 1 -- 172.21.15.88:0/1457763085 shutdown_connections 2024-03-29T10:44:24.172 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:24.189+0000 7f89b77fe640 1 -- 172.21.15.88:0/1457763085 wait complete. 2024-03-29T10:44:24.172 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:24.189+0000 7f89b77fe640 1 librados: shutdown 2024-03-29T10:44:24.188 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:44:30.995 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:44:31.145 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:31.160+0000 7f51658ac640 10 monclient: get_monmap_and_config 2024-03-29T10:44:31.145 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:31.160+0000 7f51658ac640 10 monclient: build_initial_monmap 2024-03-29T10:44:31.145 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:31.160+0000 7f51658ac640 10 monclient: monmap: 2024-03-29T10:44:31.145 INFO:teuthology.misc.health.smithi088.stderr:epoch 0 2024-03-29T10:44:31.145 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:44:31.145 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:44:31.161680+0000 2024-03-29T10:44:31.145 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:44:31.161680+0000 2024-03-29T10:44:31.146 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 0 (unknown) 2024-03-29T10:44:31.146 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:44:31.146 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-03-29T10:44:31.146 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-29T10:44:31.146 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-b 2024-03-29T10:44:31.146 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:44:31.146 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:31.163+0000 7f51658ac640 1 Processor -- start 2024-03-29T10:44:31.147 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:31.163+0000 7f51658ac640 1 -- start start 2024-03-29T10:44:31.147 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:31.163+0000 7f51658ac640 10 monclient: init 2024-03-29T10:44:31.157 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:31.164+0000 7f51658ac640 10 monclient: _reopen_session rank -1 2024-03-29T10:44:31.157 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:31.164+0000 7f51658ac640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:44:31.157 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:31.164+0000 7f51658ac640 10 monclient(hunting): picked mon.noname-a con 0x7f5158006410 addr v1:172.21.15.88:6789/0 2024-03-29T10:44:31.157 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:31.164+0000 7f51658ac640 10 monclient(hunting): picked mon.noname-b con 0x7f515802a640 addr v1:172.21.15.88:6790/0 2024-03-29T10:44:31.158 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:31.164+0000 7f51658ac640 10 monclient(hunting): picked mon.noname-c con 0x7f515802b370 addr v1:172.21.15.196:6789/0 2024-03-29T10:44:31.158 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:31.164+0000 7f51658ac640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f515802a120 con 0x7f5158006410 2024-03-29T10:44:31.158 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:31.164+0000 7f51658ac640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51580c75b0 con 0x7f515802b370 2024-03-29T10:44:31.158 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:31.164+0000 7f51658ac640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5158033110 con 0x7f515802a640 2024-03-29T10:44:31.158 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:31.164+0000 7f51658ac640 10 monclient(hunting): _renew_subs 2024-03-29T10:44:31.159 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:31.164+0000 7f51658ac640 10 monclient(hunting): authenticate will time out at 993.400666s 2024-03-29T10:44:31.159 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:31.167+0000 7f51648aa640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f5158006410 0x7f51580345e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:50029/0 (socket says 172.21.15.88:50029) 2024-03-29T10:44:31.159 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:31.167+0000 7f51648aa640 1 -- 172.21.15.88:0/461625620 learned_addr learned my addr 172.21.15.88:0/461625620 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-29T10:44:31.160 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:31.168+0000 7f515f7fe640 1 -- 172.21.15.88:0/461625620 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1940847440 0 0) 0x7f51580c75b0 con 0x7f515802b370 2024-03-29T10:44:31.161 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:31.168+0000 7f515f7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:44:31.162 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:31.168+0000 7f515f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:31.162 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:31.168+0000 7f515f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:31.162 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:31.168+0000 7f515f7fe640 10 monclient(hunting): my global_id is 4351 2024-03-29T10:44:31.163 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:31.168+0000 7f515f7fe640 1 -- 172.21.15.88:0/461625620 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5150003520 con 0x7f515802b370 2024-03-29T10:44:31.163 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:31.168+0000 7f515f7fe640 1 -- 172.21.15.88:0/461625620 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1811270211 0 0) 0x7f515802a120 con 0x7f5158006410 2024-03-29T10:44:31.164 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:31.168+0000 7f515f7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:44:31.164 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:31.168+0000 7f515f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:31.164 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:31.168+0000 7f515f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:31.164 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:31.168+0000 7f515f7fe640 10 monclient(hunting): my global_id is 4356 2024-03-29T10:44:31.165 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:31.168+0000 7f515f7fe640 1 -- 172.21.15.88:0/461625620 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f51580c75b0 con 0x7f5158006410 2024-03-29T10:44:31.165 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:31.168+0000 7f515f7fe640 1 -- 172.21.15.88:0/461625620 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2752578839 0 0) 0x7f5150003520 con 0x7f515802b370 2024-03-29T10:44:31.165 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:31.169+0000 7f515f7fe640 1 -- 172.21.15.88:0/461625620 >> v1:172.21.15.88:6790/0 conn(0x7f515802a640 legacy=0x7f51580c6ea0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:44:31.165 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:31.173+0000 7f515f7fe640 1 -- 172.21.15.88:0/461625620 >> v1:172.21.15.88:6789/0 conn(0x7f5158006410 legacy=0x7f51580345e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:31.165 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:31.173+0000 7f515f7fe640 10 monclient: _finish_hunting 0 2024-03-29T10:44:31.165 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:31.173+0000 7f515f7fe640 1 monclient: found mon.noname-c 2024-03-29T10:44:31.165 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:31.173+0000 7f515f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:44:31.166 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:31.173+0000 7f515f7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.196:6789/0 2024-03-29T10:44:31.166 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:31.173+0000 7f515f7fe640 1 -- 172.21.15.88:0/461625620 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51580332a0 con 0x7f515802b370 2024-03-29T10:44:31.166 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:31.173+0000 7f515f7fe640 10 monclient: _finish_auth 0 2024-03-29T10:44:31.166 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:31.173+0000 7f515f7fe640 10 monclient: _check_auth_tickets 2024-03-29T10:44:31.167 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:31.173+0000 7f515f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:44:31.167 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:31.173+0000 7f515f7fe640 1 -- 172.21.15.88:0/461625620 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f5154001690 con 0x7f515802b370 2024-03-29T10:44:31.168 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:31.173+0000 7f515f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:31.168 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:31.173+0000 7f515f7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:44:31.169 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:31.173+0000 7f515f7fe640 10 monclient: dump: 2024-03-29T10:44:31.169 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:44:31.169 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:44:31.171 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:44:31.171 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:44:31.171 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:31.171 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:44:31.171 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:44:31.171 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:44:31.171 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:44:31.171 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:44:31.171 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:31.173+0000 7f51658ac640 5 monclient: authenticate success, global_id 4351 2024-03-29T10:44:31.171 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:31.173+0000 7f51658ac640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:44:31.178 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:31.176+0000 7f515f7fe640 1 -- 172.21.15.88:0/461625620 <== mon.1 v1:172.21.15.196:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f51540017f0 con 0x7f515802b370 2024-03-29T10:44:31.178 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:31.176+0000 7f515f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:44:31.178 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:31.176+0000 7f515f7fe640 1 -- 172.21.15.88:0/461625620 <== mon.1 v1:172.21.15.196:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f5154001bc0 con 0x7f515802b370 2024-03-29T10:44:31.178 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:31.176+0000 7f515f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:31.178 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:31.176+0000 7f515f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:44:31.178 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:31.176+0000 7f515f7fe640 10 monclient: dump: 2024-03-29T10:44:31.178 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:44:31.178 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:44:31.179 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:44:31.179 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:44:31.179 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:31.179 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:44:31.179 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:44:31.179 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:44:31.179 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:44:31.179 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:44:31.179 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:31.176+0000 7f51658ac640 10 monclient: get_monmap_and_config success 2024-03-29T10:44:31.179 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:31.176+0000 7f51658ac640 4 set_mon_vals no callback set 2024-03-29T10:44:31.179 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:31.177+0000 7f51658ac640 10 monclient: shutdown 2024-03-29T10:44:31.179 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:31.177+0000 7f51658ac640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:44:31.179 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:31.177+0000 7f51658ac640 1 -- 172.21.15.88:0/461625620 >> v1:172.21.15.196:6789/0 conn(0x7f515802b370 legacy=0x7f51580320c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:31.179 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:31.177+0000 7f51658ac640 1 -- 172.21.15.88:0/461625620 shutdown_connections 2024-03-29T10:44:31.180 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:31.177+0000 7f51658ac640 1 -- 172.21.15.88:0/461625620 >> 172.21.15.88:0/461625620 conn(0x7f51580058f0 msgr2=0x7f5158005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:44:31.180 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:31.178+0000 7f51658ac640 1 -- 172.21.15.88:0/461625620 shutdown_connections 2024-03-29T10:44:31.180 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:31.178+0000 7f51658ac640 1 -- 172.21.15.88:0/461625620 wait complete. 2024-03-29T10:44:31.180 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:31.179+0000 7f51658ac640 10 monclient: build_initial_monmap 2024-03-29T10:44:31.180 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:31.179+0000 7f51658ac640 10 monclient: monmap: 2024-03-29T10:44:31.180 INFO:teuthology.misc.health.smithi088.stderr:epoch 0 2024-03-29T10:44:31.180 INFO:teuthology.misc.health.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:44:31.180 INFO:teuthology.misc.health.smithi088.stderr:last_changed 0.000000 2024-03-29T10:44:31.180 INFO:teuthology.misc.health.smithi088.stderr:created 0.000000 2024-03-29T10:44:31.180 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 0 (unknown) 2024-03-29T10:44:31.180 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:44:31.180 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-03-29T10:44:31.180 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-29T10:44:31.180 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-c 2024-03-29T10:44:31.181 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:44:31.181 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:31.179+0000 7f51658ac640 1 librados: starting msgr at 2024-03-29T10:44:31.181 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:31.179+0000 7f51658ac640 1 librados: starting objecter 2024-03-29T10:44:31.181 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:31.179+0000 7f51658ac640 1 Processor -- start 2024-03-29T10:44:31.181 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:31.179+0000 7f51658ac640 1 -- start start 2024-03-29T10:44:31.181 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:31.179+0000 7f51658ac640 1 librados: setting wanted keys 2024-03-29T10:44:31.181 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:31.179+0000 7f51658ac640 1 librados: calling monclient init 2024-03-29T10:44:31.181 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:31.179+0000 7f51658ac640 10 monclient: init 2024-03-29T10:44:31.181 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:31.180+0000 7f51658ac640 10 monclient: _reopen_session rank -1 2024-03-29T10:44:31.181 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:31.180+0000 7f51658ac640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:44:31.181 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:31.180+0000 7f51658ac640 10 monclient(hunting): picked mon.noname-b con 0x7f5158006410 addr v1:172.21.15.196:6789/0 2024-03-29T10:44:31.181 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:31.180+0000 7f51658ac640 10 monclient(hunting): picked mon.noname-c con 0x7f515802a640 addr v1:172.21.15.88:6790/0 2024-03-29T10:44:31.181 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:31.180+0000 7f51658ac640 10 monclient(hunting): picked mon.noname-a con 0x7f515802b370 addr v1:172.21.15.88:6789/0 2024-03-29T10:44:31.181 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:31.180+0000 7f51658ac640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51580c28c0 con 0x7f515802b370 2024-03-29T10:44:31.182 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:31.180+0000 7f51658ac640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51580c2af0 con 0x7f5158006410 2024-03-29T10:44:31.182 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:31.180+0000 7f51658ac640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51580c1720 con 0x7f515802a640 2024-03-29T10:44:31.182 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:31.180+0000 7f51658ac640 10 monclient(hunting): _renew_subs 2024-03-29T10:44:31.182 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:31.180+0000 7f51658ac640 10 monclient(hunting): authenticate will time out at 993.416754s 2024-03-29T10:44:31.182 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:31.184+0000 7f515ffff640 1 --1- >> v1:172.21.15.88:6790/0 conn(0x7f515802a640 0x7f5158168730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6790/0 says I am v1:172.21.15.88:37971/0 (socket says 172.21.15.88:37971) 2024-03-29T10:44:31.182 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:31.184+0000 7f515ffff640 1 -- 172.21.15.88:0/1766028997 learned_addr learned my addr 172.21.15.88:0/1766028997 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-29T10:44:31.182 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:31.185+0000 7f515d7fa640 10 client.?.objecter ms_handle_connect 0x7f515802a640 2024-03-29T10:44:31.182 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:31.185+0000 7f515d7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:44:31.182 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:31.185+0000 7f515d7fa640 1 -- 172.21.15.88:0/1766028997 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 109316150 0 0) 0x7f51580c1720 con 0x7f515802a640 2024-03-29T10:44:31.182 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:31.185+0000 7f515d7fa640 10 monclient(hunting): _negotiate 2024-03-29T10:44:31.182 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:31.185+0000 7f515d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:31.182 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:31.185+0000 7f515d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:31.182 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:31.185+0000 7f515d7fa640 10 monclient(hunting): my global_id is 4379 2024-03-29T10:44:31.183 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:31.185+0000 7f515d7fa640 1 -- 172.21.15.88:0/1766028997 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5144004eb0 con 0x7f515802a640 2024-03-29T10:44:31.183 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:31.185+0000 7f515d7fa640 1 -- 172.21.15.88:0/1766028997 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1647854345 0 0) 0x7f5144004eb0 con 0x7f515802a640 2024-03-29T10:44:31.183 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:31.185+0000 7f515d7fa640 1 -- 172.21.15.88:0/1766028997 --> v1:172.21.15.88:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f51580c1720 con 0x7f515802a640 2024-03-29T10:44:31.183 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:31.186+0000 7f515d7fa640 1 -- 172.21.15.88:0/1766028997 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f516004fdd0 con 0x7f515802a640 2024-03-29T10:44:31.183 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:31.186+0000 7f515d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:31.183 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:31.186+0000 7f515d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:44:31.183 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:31.186+0000 7f515d7fa640 10 monclient(hunting): dump: 2024-03-29T10:44:31.183 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:44:31.183 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:44:31.183 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:44:31.183 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:44:31.183 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:31.183 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:44:31.183 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:44:31.183 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:44:31.184 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:44:31.184 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:44:31.184 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:31.186+0000 7f515d7fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:44:31.184 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:31.186+0000 7f515d7fa640 1 -- 172.21.15.88:0/1766028997 <== mon.2 v1:172.21.15.88:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3276644896 0 0) 0x7f51580c1720 con 0x7f515802a640 2024-03-29T10:44:31.184 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:31.186+0000 7f515d7fa640 1 -- 172.21.15.88:0/1766028997 >> v1:172.21.15.196:6789/0 conn(0x7f5158006410 legacy=0x7f51580c2f10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:44:31.184 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:31.187+0000 7f515d7fa640 1 -- 172.21.15.88:0/1766028997 >> v1:172.21.15.88:6789/0 conn(0x7f515802b370 legacy=0x7f515816be60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:31.184 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:31.187+0000 7f515d7fa640 10 monclient: _finish_hunting 0 2024-03-29T10:44:31.184 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:31.187+0000 7f515d7fa640 1 monclient: found mon.c 2024-03-29T10:44:31.184 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:31.187+0000 7f515d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:44:31.184 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:31.187+0000 7f515d7fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-03-29T10:44:31.184 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:31.187+0000 7f515d7fa640 1 -- 172.21.15.88:0/1766028997 --> v1:172.21.15.88:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51580c18b0 con 0x7f515802a640 2024-03-29T10:44:31.184 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:31.187+0000 7f515d7fa640 10 monclient: _finish_auth 0 2024-03-29T10:44:31.184 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:31.187+0000 7f515d7fa640 10 monclient: _check_auth_tickets 2024-03-29T10:44:31.184 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:31.187+0000 7f515d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:44:31.185 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:31.187+0000 7f51658ac640 5 monclient: authenticate success, global_id 4379 2024-03-29T10:44:31.185 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:31.187+0000 7f51658ac640 10 monclient: _renew_subs 2024-03-29T10:44:31.185 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:31.187+0000 7f51658ac640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-03-29T10:44:31.185 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:31.187+0000 7f51658ac640 1 -- 172.21.15.88:0/1766028997 --> v1:172.21.15.88:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51580c0aa0 con 0x7f515802a640 2024-03-29T10:44:31.185 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:31.187+0000 7f51658ac640 10 client.4379.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:44:31.185 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:31.187+0000 7f51658ac640 10 monclient: _renew_subs 2024-03-29T10:44:31.185 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:31.187+0000 7f51658ac640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-03-29T10:44:31.185 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:31.187+0000 7f51658ac640 1 -- 172.21.15.88:0/1766028997 --> v1:172.21.15.88:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f51580c3800 con 0x7f515802a640 2024-03-29T10:44:31.185 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:31.187+0000 7f51658ac640 1 librados: init done 2024-03-29T10:44:31.185 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:31.188+0000 7f515d7fa640 1 -- 172.21.15.88:0/1766028997 <== mon.2 v1:172.21.15.88:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f516004f400 con 0x7f515802a640 2024-03-29T10:44:31.185 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:31.188+0000 7f515d7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:44:31.185 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:31.188+0000 7f515d7fa640 1 -- 172.21.15.88:0/1766028997 <== mon.2 v1:172.21.15.88:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f5160075f10 con 0x7f515802a640 2024-03-29T10:44:31.185 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:31.188+0000 7f515d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:31.185 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:31.188+0000 7f515d7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:44:31.186 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:31.188+0000 7f515d7fa640 10 monclient: dump: 2024-03-29T10:44:31.186 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:44:31.186 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:44:31.186 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:44:31.186 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:44:31.186 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:31.186 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:44:31.186 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:44:31.186 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:44:31.186 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:44:31.186 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:44:31.186 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:31.188+0000 7f515effd640 4 set_mon_vals no callback set 2024-03-29T10:44:31.186 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:31.190+0000 7f51658ac640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:44:31.186 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:31.190+0000 7f51658ac640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:44:31.187 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:31.190+0000 7f51658ac640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-03-29T10:44:31.187 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:31.190+0000 7f51658ac640 1 -- 172.21.15.88:0/1766028997 --> v1:172.21.15.88:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5140005080 con 0x7f515802a640 2024-03-29T10:44:31.187 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:31.190+0000 7f515d7fa640 1 -- 172.21.15.88:0/1766028997 <== mon.2 v1:172.21.15.88:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4235629220 0 0) 0x7f5160067bf0 con 0x7f515802a640 2024-03-29T10:44:31.187 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:31.191+0000 7f515d7fa640 1 -- 172.21.15.88:0/1766028997 <== mon.2 v1:172.21.15.88:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4257673582 0 0) 0x7f51600ff680 con 0x7f515802a640 2024-03-29T10:44:31.187 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:31.191+0000 7f515d7fa640 10 client.4379.objecter ms_dispatch 0x7f5160000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:44:31.187 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:31.191+0000 7f515d7fa640 3 client.4379.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:44:31.187 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:31.191+0000 7f515d7fa640 3 client.4379.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:44:31.187 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:31.191+0000 7f515d7fa640 20 client.4379.objecter dump_active .. 0 homeless 2024-03-29T10:44:31.187 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:31.191+0000 7f51648aa640 10 monclient: get_auth_request con 0x7f514406db10 auth_method 0 2024-03-29T10:44:31.187 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:31.191+0000 7f515d7fa640 10 client.4379.objecter ms_handle_connect 0x7f514406db10 2024-03-29T10:44:31.188 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:31.198+0000 7f515d7fa640 1 -- 172.21.15.88:0/1766028997 <== mon.2 v1:172.21.15.88:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f51600cafc0 con 0x7f515802a640 2024-03-29T10:44:31.188 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:31.198+0000 7f515d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:44:31.188 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:31.198+0000 7f515d7fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:44:31.504 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:31.521+0000 7f51658ac640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:44:31.504 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:31.521+0000 7f51658ac640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:44:31.504 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:31.521+0000 7f51658ac640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-03-29T10:44:31.504 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:31.521+0000 7f51658ac640 1 -- 172.21.15.88:0/1766028997 --> v1:172.21.15.88:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5140005330 con 0x7f515802a640 2024-03-29T10:44:31.505 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:31.522+0000 7f515d7fa640 1 -- 172.21.15.88:0/1766028997 <== mon.2 v1:172.21.15.88:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 344506374) 0x7f51600ccef0 con 0x7f515802a640 2024-03-29T10:44:31.505 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:31.522+0000 7f515d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:44:31.505 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:31.522+0000 7f515d7fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:44:31.505 INFO:teuthology.misc.health.smithi088.stdout: 2024-03-29T10:44:31.505 INFO:teuthology.misc.health.smithi088.stdout:{"epoch":10,"fsid":"869f5188-f92b-42dd-a46d-8c646e0388a9","created":"2024-03-29T10:40:40.285087+0000","modified":"2024-03-29T10:40:42.536230+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:40:42.536230+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a26c0c1c-8de0-438c-a3f6-3df10ea61574","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5de24915-8dec-49b6-8b16-f3170197c591","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ac054af3-d589-4565-9644-e6af75975c53","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3ec4f045-c714-4a22-bd1a-e07330b45ab3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"cddb9867-8e30-4951-9b13-3d1e78b9c45a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dc6d9911-785d-432f-8b53-56a1ccab302f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"77c88437-b587-496e-8b4d-1824ea8b5305","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ce963d96-5a8e-44c3-99de-c0dbebe234ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:44:31.511 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:31.526+0000 7f51658ac640 10 librados: watch_flush enter 2024-03-29T10:44:31.511 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:31.526+0000 7f51658ac640 10 librados: watch_flush exit 2024-03-29T10:44:31.511 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:31.526+0000 7f51658ac640 20 client.4379.objecter shutdown clearing up homeless session... 2024-03-29T10:44:31.511 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:31.526+0000 7f51658ac640 10 client.4379.objecter successfully canceled tick 2024-03-29T10:44:31.512 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:31.527+0000 7f51658ac640 1 -- 172.21.15.88:0/1766028997 >> v1:172.21.15.88:6816/1752861349 conn(0x7f514406db10 legacy=0x7f514406ffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:31.512 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:31.527+0000 7f51658ac640 10 monclient: shutdown 2024-03-29T10:44:31.512 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:31.527+0000 7f51658ac640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:44:31.512 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:31.527+0000 7f51658ac640 1 -- 172.21.15.88:0/1766028997 >> v1:172.21.15.88:6790/0 conn(0x7f515802a640 legacy=0x7f5158168730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:31.512 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:31.527+0000 7f51658ac640 1 -- 172.21.15.88:0/1766028997 shutdown_connections 2024-03-29T10:44:31.512 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:31.527+0000 7f51658ac640 1 -- 172.21.15.88:0/1766028997 >> 172.21.15.88:0/1766028997 conn(0x7f51580058f0 msgr2=0x7f51580043d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:44:31.512 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:31.529+0000 7f51658ac640 1 -- 172.21.15.88:0/1766028997 shutdown_connections 2024-03-29T10:44:31.513 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:31.530+0000 7f51658ac640 1 -- 172.21.15.88:0/1766028997 wait complete. 2024-03-29T10:44:31.513 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:31.530+0000 7f51658ac640 1 librados: shutdown 2024-03-29T10:44:31.531 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:44:38.338 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:44:38.489 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:38.506+0000 7f2f53225640 10 monclient: get_monmap_and_config 2024-03-29T10:44:38.490 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:38.506+0000 7f2f53225640 10 monclient: build_initial_monmap 2024-03-29T10:44:38.490 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:38.506+0000 7f2f53225640 10 monclient: monmap: 2024-03-29T10:44:38.490 INFO:teuthology.misc.health.smithi088.stderr:epoch 0 2024-03-29T10:44:38.490 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:44:38.490 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:44:38.507641+0000 2024-03-29T10:44:38.490 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:44:38.507641+0000 2024-03-29T10:44:38.490 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 0 (unknown) 2024-03-29T10:44:38.490 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:44:38.490 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-03-29T10:44:38.490 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-29T10:44:38.490 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-b 2024-03-29T10:44:38.490 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:44:38.491 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:38.508+0000 7f2f53225640 1 Processor -- start 2024-03-29T10:44:38.492 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:38.509+0000 7f2f53225640 1 -- start start 2024-03-29T10:44:38.492 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:38.509+0000 7f2f53225640 10 monclient: init 2024-03-29T10:44:38.493 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:38.510+0000 7f2f53225640 10 monclient: _reopen_session rank -1 2024-03-29T10:44:38.493 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:38.510+0000 7f2f53225640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:44:38.493 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:38.510+0000 7f2f53225640 10 monclient(hunting): picked mon.noname-a con 0x7f2f4c1457f0 addr v1:172.21.15.88:6789/0 2024-03-29T10:44:38.493 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:38.510+0000 7f2f53225640 10 monclient(hunting): picked mon.noname-b con 0x7f2f4c13f650 addr v1:172.21.15.88:6790/0 2024-03-29T10:44:38.496 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:38.513+0000 7f2f53225640 10 monclient(hunting): picked mon.noname-c con 0x7f2f4c0a9d70 addr v1:172.21.15.196:6789/0 2024-03-29T10:44:38.496 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:38.513+0000 7f2f53225640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f4c0aaef0 con 0x7f2f4c1457f0 2024-03-29T10:44:38.497 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:38.513+0000 7f2f53225640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f4c145bd0 con 0x7f2f4c0a9d70 2024-03-29T10:44:38.497 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:38.513+0000 7f2f53225640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f4c145eb0 con 0x7f2f4c13f650 2024-03-29T10:44:38.497 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:38.513+0000 7f2f53225640 10 monclient(hunting): _renew_subs 2024-03-29T10:44:38.497 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:38.513+0000 7f2f53225640 10 monclient(hunting): authenticate will time out at 1000.749875s 2024-03-29T10:44:38.500 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:38.516+0000 7f2f4bfff640 1 --1- >> v1:172.21.15.88:6790/0 conn(0x7f2f4c13f650 0x7f2f4c0b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6790/0 says I am v1:172.21.15.88:40285/0 (socket says 172.21.15.88:40285) 2024-03-29T10:44:38.500 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:38.516+0000 7f2f4bfff640 1 -- 172.21.15.88:0/3494595885 learned_addr learned my addr 172.21.15.88:0/3494595885 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-29T10:44:38.500 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:38.517+0000 7f2f4b7fe640 1 -- 172.21.15.88:0/3494595885 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2430156669 0 0) 0x7f2f4c0aaef0 con 0x7f2f4c1457f0 2024-03-29T10:44:38.500 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:38.517+0000 7f2f4b7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:44:38.500 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:38.517+0000 7f2f4b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:38.501 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:38.517+0000 7f2f4b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:38.501 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:38.517+0000 7f2f4b7fe640 10 monclient(hunting): my global_id is 4359 2024-03-29T10:44:38.501 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:38.517+0000 7f2f4b7fe640 1 -- 172.21.15.88:0/3494595885 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2f34003520 con 0x7f2f4c1457f0 2024-03-29T10:44:38.501 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:38.517+0000 7f2f4b7fe640 1 -- 172.21.15.88:0/3494595885 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1897970450 0 0) 0x7f2f4c145bd0 con 0x7f2f4c0a9d70 2024-03-29T10:44:38.501 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:38.517+0000 7f2f4b7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:44:38.501 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:38.517+0000 7f2f4b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:38.501 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:38.517+0000 7f2f4b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:38.501 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:38.517+0000 7f2f4b7fe640 10 monclient(hunting): my global_id is 4354 2024-03-29T10:44:38.501 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:38.517+0000 7f2f4b7fe640 1 -- 172.21.15.88:0/3494595885 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2f34003840 con 0x7f2f4c0a9d70 2024-03-29T10:44:38.502 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:38.518+0000 7f2f4b7fe640 1 -- 172.21.15.88:0/3494595885 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3010130444 0 0) 0x7f2f34003840 con 0x7f2f4c0a9d70 2024-03-29T10:44:38.502 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:38.518+0000 7f2f4b7fe640 1 -- 172.21.15.88:0/3494595885 >> v1:172.21.15.88:6790/0 conn(0x7f2f4c13f650 legacy=0x7f2f4c0b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:38.502 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:38.518+0000 7f2f4b7fe640 1 -- 172.21.15.88:0/3494595885 >> v1:172.21.15.88:6789/0 conn(0x7f2f4c1457f0 legacy=0x7f2f4c13ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:38.502 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:38.518+0000 7f2f4b7fe640 10 monclient: _finish_hunting 0 2024-03-29T10:44:38.502 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:38.518+0000 7f2f4b7fe640 1 monclient: found mon.noname-c 2024-03-29T10:44:38.502 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:38.518+0000 7f2f4b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:44:38.502 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:38.518+0000 7f2f4b7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.196:6789/0 2024-03-29T10:44:38.502 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:38.518+0000 7f2f4b7fe640 1 -- 172.21.15.88:0/3494595885 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f4c146340 con 0x7f2f4c0a9d70 2024-03-29T10:44:38.502 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:38.518+0000 7f2f4b7fe640 10 monclient: _finish_auth 0 2024-03-29T10:44:38.503 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:38.518+0000 7f2f4b7fe640 10 monclient: _check_auth_tickets 2024-03-29T10:44:38.503 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:38.518+0000 7f2f4b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:44:38.503 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:38.518+0000 7f2f4b7fe640 1 -- 172.21.15.88:0/3494595885 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f2f3c001670 con 0x7f2f4c0a9d70 2024-03-29T10:44:38.503 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:38.518+0000 7f2f4b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:38.503 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:38.518+0000 7f2f4b7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:44:38.503 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:38.518+0000 7f2f4b7fe640 10 monclient: dump: 2024-03-29T10:44:38.503 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:44:38.503 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:44:38.503 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:44:38.503 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:44:38.503 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:38.503 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:44:38.503 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:44:38.503 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:44:38.504 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:44:38.504 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:44:38.504 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:38.518+0000 7f2f53225640 5 monclient: authenticate success, global_id 4354 2024-03-29T10:44:38.504 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:38.518+0000 7f2f53225640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:44:38.504 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:38.518+0000 7f2f4b7fe640 1 -- 172.21.15.88:0/3494595885 <== mon.1 v1:172.21.15.196:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2f3c0017d0 con 0x7f2f4c0a9d70 2024-03-29T10:44:38.504 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:38.518+0000 7f2f4b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:44:38.504 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:38.518+0000 7f2f4b7fe640 1 -- 172.21.15.88:0/3494595885 <== mon.1 v1:172.21.15.196:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f2f3c001be0 con 0x7f2f4c0a9d70 2024-03-29T10:44:38.504 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:38.518+0000 7f2f4b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:38.504 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:38.518+0000 7f2f4b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:44:38.504 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:38.518+0000 7f2f4b7fe640 10 monclient: dump: 2024-03-29T10:44:38.504 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:44:38.504 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:44:38.504 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:44:38.504 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:44:38.505 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:38.505 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:44:38.505 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:44:38.505 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:44:38.505 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:44:38.505 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:44:38.505 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:38.518+0000 7f2f53225640 10 monclient: get_monmap_and_config success 2024-03-29T10:44:38.505 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:38.518+0000 7f2f53225640 4 set_mon_vals no callback set 2024-03-29T10:44:38.505 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:38.519+0000 7f2f53225640 10 monclient: shutdown 2024-03-29T10:44:38.505 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:38.519+0000 7f2f53225640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:44:38.505 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:38.519+0000 7f2f53225640 1 -- 172.21.15.88:0/3494595885 >> v1:172.21.15.196:6789/0 conn(0x7f2f4c0a9d70 legacy=0x7f2f4c14d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:38.506 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:38.520+0000 7f2f53225640 1 -- 172.21.15.88:0/3494595885 shutdown_connections 2024-03-29T10:44:38.506 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:38.520+0000 7f2f53225640 1 -- 172.21.15.88:0/3494595885 >> 172.21.15.88:0/3494595885 conn(0x7f2f4c066a00 msgr2=0x7f2f4c066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:44:38.506 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:38.520+0000 7f2f53225640 1 -- 172.21.15.88:0/3494595885 shutdown_connections 2024-03-29T10:44:38.506 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:38.520+0000 7f2f53225640 1 -- 172.21.15.88:0/3494595885 wait complete. 2024-03-29T10:44:38.506 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:38.521+0000 7f2f53225640 10 monclient: build_initial_monmap 2024-03-29T10:44:38.506 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:38.521+0000 7f2f53225640 10 monclient: monmap: 2024-03-29T10:44:38.507 INFO:teuthology.misc.health.smithi088.stderr:epoch 0 2024-03-29T10:44:38.507 INFO:teuthology.misc.health.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:44:38.507 INFO:teuthology.misc.health.smithi088.stderr:last_changed 0.000000 2024-03-29T10:44:38.507 INFO:teuthology.misc.health.smithi088.stderr:created 0.000000 2024-03-29T10:44:38.507 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 0 (unknown) 2024-03-29T10:44:38.507 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:44:38.507 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-03-29T10:44:38.507 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-29T10:44:38.507 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-c 2024-03-29T10:44:38.507 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:44:38.507 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:38.521+0000 7f2f53225640 1 librados: starting msgr at 2024-03-29T10:44:38.507 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:38.521+0000 7f2f53225640 1 librados: starting objecter 2024-03-29T10:44:38.507 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:38.522+0000 7f2f53225640 1 Processor -- start 2024-03-29T10:44:38.507 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:38.522+0000 7f2f53225640 1 -- start start 2024-03-29T10:44:38.507 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:38.522+0000 7f2f53225640 1 librados: setting wanted keys 2024-03-29T10:44:38.508 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:38.522+0000 7f2f53225640 1 librados: calling monclient init 2024-03-29T10:44:38.508 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:38.522+0000 7f2f53225640 10 monclient: init 2024-03-29T10:44:38.508 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:38.522+0000 7f2f53225640 10 monclient: _reopen_session rank -1 2024-03-29T10:44:38.508 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:38.523+0000 7f2f53225640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:44:38.511 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:38.527+0000 7f2f53225640 10 monclient(hunting): picked mon.noname-b con 0x7f2f4c0a9d70 addr v1:172.21.15.196:6789/0 2024-03-29T10:44:38.511 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:38.527+0000 7f2f53225640 10 monclient(hunting): picked mon.noname-a con 0x7f2f4c13f650 addr v1:172.21.15.88:6789/0 2024-03-29T10:44:38.511 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:38.527+0000 7f2f53225640 10 monclient(hunting): picked mon.noname-c con 0x7f2f4c1457f0 addr v1:172.21.15.88:6790/0 2024-03-29T10:44:38.511 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:38.527+0000 7f2f53225640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f4c0ae580 con 0x7f2f4c13f650 2024-03-29T10:44:38.511 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:38.527+0000 7f2f53225640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f4c142c90 con 0x7f2f4c0a9d70 2024-03-29T10:44:38.511 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:38.527+0000 7f2f53225640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f4c0ad5f0 con 0x7f2f4c1457f0 2024-03-29T10:44:38.511 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:38.527+0000 7f2f53225640 10 monclient(hunting): _renew_subs 2024-03-29T10:44:38.511 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:38.527+0000 7f2f53225640 10 monclient(hunting): authenticate will time out at 1000.763169s 2024-03-29T10:44:38.520 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:38.532+0000 7f2f50fc1640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f2f4c0a9d70 0x7f2f4c0b0df0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.88:47729/0 (socket says 172.21.15.88:47729) 2024-03-29T10:44:38.520 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:38.532+0000 7f2f50fc1640 1 -- 172.21.15.88:0/2776229174 learned_addr learned my addr 172.21.15.88:0/2776229174 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-29T10:44:38.520 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:38.532+0000 7f2f497fa640 10 client.?.objecter ms_handle_connect 0x7f2f4c1457f0 2024-03-29T10:44:38.520 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:38.532+0000 7f2f497fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:44:38.520 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:38.532+0000 7f2f497fa640 10 client.?.objecter ms_handle_connect 0x7f2f4c0a9d70 2024-03-29T10:44:38.520 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:38.532+0000 7f2f497fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:44:38.520 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:38.532+0000 7f2f497fa640 1 -- 172.21.15.88:0/2776229174 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 587793196 0 0) 0x7f2f4c142c90 con 0x7f2f4c0a9d70 2024-03-29T10:44:38.520 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:38.532+0000 7f2f497fa640 10 monclient(hunting): _negotiate 2024-03-29T10:44:38.521 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:38.532+0000 7f2f497fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:38.521 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:38.532+0000 7f2f497fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:38.521 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:38.532+0000 7f2f497fa640 10 monclient(hunting): my global_id is 4357 2024-03-29T10:44:38.521 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:38.533+0000 7f2f497fa640 1 -- 172.21.15.88:0/2776229174 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2f38004eb0 con 0x7f2f4c0a9d70 2024-03-29T10:44:38.521 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:38.533+0000 7f2f497fa640 1 -- 172.21.15.88:0/2776229174 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1788300182 0 0) 0x7f2f4c0ad5f0 con 0x7f2f4c1457f0 2024-03-29T10:44:38.521 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:38.533+0000 7f2f497fa640 10 monclient(hunting): _negotiate 2024-03-29T10:44:38.521 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:38.533+0000 7f2f497fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:38.521 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:38.533+0000 7f2f497fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:38.521 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:38.533+0000 7f2f497fa640 10 monclient(hunting): my global_id is 4385 2024-03-29T10:44:38.521 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:38.533+0000 7f2f497fa640 1 -- 172.21.15.88:0/2776229174 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2f4c142c90 con 0x7f2f4c1457f0 2024-03-29T10:44:38.521 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:38.533+0000 7f2f497fa640 1 -- 172.21.15.88:0/2776229174 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4075970810 0 0) 0x7f2f38004eb0 con 0x7f2f4c0a9d70 2024-03-29T10:44:38.521 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:38.533+0000 7f2f497fa640 1 -- 172.21.15.88:0/2776229174 --> v1:172.21.15.196:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2f4c0ad5f0 con 0x7f2f4c0a9d70 2024-03-29T10:44:38.521 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:38.533+0000 7f2f497fa640 1 -- 172.21.15.88:0/2776229174 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f2f44001cc0 con 0x7f2f4c0a9d70 2024-03-29T10:44:38.521 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:38.533+0000 7f2f497fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:38.522 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:38.533+0000 7f2f497fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:44:38.522 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:38.533+0000 7f2f497fa640 10 monclient(hunting): dump: 2024-03-29T10:44:38.522 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:44:38.522 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:44:38.522 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:44:38.522 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:44:38.522 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:38.522 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:44:38.522 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:44:38.522 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:44:38.522 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:44:38.522 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:44:38.522 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:38.533+0000 7f2f497fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:44:38.522 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:38.533+0000 7f2f497fa640 1 -- 172.21.15.88:0/2776229174 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1333979961 0 0) 0x7f2f4c142c90 con 0x7f2f4c1457f0 2024-03-29T10:44:38.523 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:38.533+0000 7f2f497fa640 1 -- 172.21.15.88:0/2776229174 --> v1:172.21.15.88:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2f38004eb0 con 0x7f2f4c1457f0 2024-03-29T10:44:38.523 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:38.534+0000 7f2f497fa640 1 -- 172.21.15.88:0/2776229174 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f2f3c001ae0 con 0x7f2f4c1457f0 2024-03-29T10:44:38.523 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:38.534+0000 7f2f497fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:38.523 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:38.534+0000 7f2f497fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:44:38.523 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:38.534+0000 7f2f497fa640 10 monclient(hunting): dump: 2024-03-29T10:44:38.523 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:44:38.523 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:44:38.523 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:44:38.523 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:44:38.523 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:38.523 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:44:38.523 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:44:38.523 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:44:38.524 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:44:38.524 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:44:38.524 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:38.534+0000 7f2f497fa640 1 -- 172.21.15.88:0/2776229174 <== mon.1 v1:172.21.15.196:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 467549575 0 0) 0x7f2f4c0ad5f0 con 0x7f2f4c0a9d70 2024-03-29T10:44:38.524 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:38.534+0000 7f2f497fa640 1 -- 172.21.15.88:0/2776229174 >> v1:172.21.15.88:6790/0 conn(0x7f2f4c1457f0 legacy=0x7f2f4c1e2ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:38.524 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:38.534+0000 7f2f497fa640 1 -- 172.21.15.88:0/2776229174 >> v1:172.21.15.88:6789/0 conn(0x7f2f4c13f650 legacy=0x7f2f4c1da720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:38.524 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:38.534+0000 7f2f497fa640 10 monclient: _finish_hunting 0 2024-03-29T10:44:38.524 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:38.534+0000 7f2f497fa640 1 monclient: found mon.b 2024-03-29T10:44:38.524 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:38.534+0000 7f2f497fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:44:38.524 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:38.534+0000 7f2f497fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T10:44:38.524 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:38.534+0000 7f2f497fa640 1 -- 172.21.15.88:0/2776229174 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f4c0ad780 con 0x7f2f4c0a9d70 2024-03-29T10:44:38.524 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:38.535+0000 7f2f497fa640 10 monclient: _finish_auth 0 2024-03-29T10:44:38.524 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:38.535+0000 7f2f497fa640 10 monclient: _check_auth_tickets 2024-03-29T10:44:38.524 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:38.535+0000 7f2f497fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:44:38.524 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:38.535+0000 7f2f53225640 5 monclient: authenticate success, global_id 4357 2024-03-29T10:44:38.525 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:38.535+0000 7f2f53225640 10 monclient: _renew_subs 2024-03-29T10:44:38.525 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:38.535+0000 7f2f53225640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T10:44:38.525 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:38.535+0000 7f2f53225640 1 -- 172.21.15.88:0/2776229174 --> v1:172.21.15.196:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f4c0adab0 con 0x7f2f4c0a9d70 2024-03-29T10:44:38.525 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:38.535+0000 7f2f53225640 10 client.4357.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:44:38.525 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:38.535+0000 7f2f53225640 10 monclient: _renew_subs 2024-03-29T10:44:38.525 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:38.535+0000 7f2f53225640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T10:44:38.525 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:38.535+0000 7f2f53225640 1 -- 172.21.15.88:0/2776229174 --> v1:172.21.15.196:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f4c0b16e0 con 0x7f2f4c0a9d70 2024-03-29T10:44:38.525 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:38.536+0000 7f2f53225640 1 librados: init done 2024-03-29T10:44:38.525 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:38.536+0000 7f2f497fa640 1 -- 172.21.15.88:0/2776229174 <== mon.1 v1:172.21.15.196:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2f44001e20 con 0x7f2f4c0a9d70 2024-03-29T10:44:38.525 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:38.536+0000 7f2f497fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:44:38.525 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:38.536+0000 7f2f497fa640 1 -- 172.21.15.88:0/2776229174 <== mon.1 v1:172.21.15.196:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f2f44003c20 con 0x7f2f4c0a9d70 2024-03-29T10:44:38.525 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:38.536+0000 7f2f497fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:38.525 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:38.536+0000 7f2f497fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:44:38.525 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:38.536+0000 7f2f497fa640 10 monclient: dump: 2024-03-29T10:44:38.525 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:44:38.526 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:44:38.526 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:44:38.526 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:44:38.526 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:38.526 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:44:38.526 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:44:38.526 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:44:38.526 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:44:38.526 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:44:38.528 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:38.537+0000 7f2f4a7fc640 4 set_mon_vals no callback set 2024-03-29T10:44:38.528 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:38.538+0000 7f2f53225640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:44:38.528 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:38.538+0000 7f2f53225640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:44:38.528 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:38.538+0000 7f2f53225640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T10:44:38.528 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:38.538+0000 7f2f53225640 1 -- 172.21.15.88:0/2776229174 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f2c005080 con 0x7f2f4c0a9d70 2024-03-29T10:44:38.528 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:38.539+0000 7f2f497fa640 1 -- 172.21.15.88:0/2776229174 <== mon.1 v1:172.21.15.196:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4235629220 0 0) 0x7f2f44003d80 con 0x7f2f4c0a9d70 2024-03-29T10:44:38.528 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:38.540+0000 7f2f497fa640 1 -- 172.21.15.88:0/2776229174 <== mon.1 v1:172.21.15.196:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4257673582 0 0) 0x7f2f44084c90 con 0x7f2f4c0a9d70 2024-03-29T10:44:38.528 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:38.540+0000 7f2f497fa640 10 client.4357.objecter ms_dispatch 0x7f2f4c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:44:38.528 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:38.540+0000 7f2f497fa640 3 client.4357.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:44:38.528 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:38.540+0000 7f2f497fa640 3 client.4357.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:44:38.528 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:38.540+0000 7f2f497fa640 20 client.4357.objecter dump_active .. 0 homeless 2024-03-29T10:44:38.528 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:38.540+0000 7f2f4bfff640 10 monclient: get_auth_request con 0x7f2f3806def0 auth_method 0 2024-03-29T10:44:38.529 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:38.541+0000 7f2f497fa640 10 client.4357.objecter ms_handle_connect 0x7f2f3806def0 2024-03-29T10:44:38.533 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:38.548+0000 7f2f497fa640 1 -- 172.21.15.88:0/2776229174 <== mon.1 v1:172.21.15.196:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2f44050570 con 0x7f2f4c0a9d70 2024-03-29T10:44:38.533 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:38.548+0000 7f2f497fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:44:38.533 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:38.548+0000 7f2f497fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:44:38.854 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:38.871+0000 7f2f53225640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:44:38.854 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:38.871+0000 7f2f53225640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:44:38.854 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:38.871+0000 7f2f53225640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T10:44:38.855 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:38.871+0000 7f2f53225640 1 -- 172.21.15.88:0/2776229174 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2f2c005330 con 0x7f2f4c0a9d70 2024-03-29T10:44:38.857 INFO:teuthology.misc.health.smithi088.stdout: 2024-03-29T10:44:38.857 INFO:teuthology.misc.health.smithi088.stdout:{"epoch":10,"fsid":"869f5188-f92b-42dd-a46d-8c646e0388a9","created":"2024-03-29T10:40:40.285087+0000","modified":"2024-03-29T10:40:42.536230+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:40:42.536230+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a26c0c1c-8de0-438c-a3f6-3df10ea61574","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5de24915-8dec-49b6-8b16-f3170197c591","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ac054af3-d589-4565-9644-e6af75975c53","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3ec4f045-c714-4a22-bd1a-e07330b45ab3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"cddb9867-8e30-4951-9b13-3d1e78b9c45a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dc6d9911-785d-432f-8b53-56a1ccab302f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"77c88437-b587-496e-8b4d-1824ea8b5305","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ce963d96-5a8e-44c3-99de-c0dbebe234ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:44:38.857 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:38.872+0000 7f2f497fa640 1 -- 172.21.15.88:0/2776229174 <== mon.1 v1:172.21.15.196:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 344506374) 0x7f2f440524a0 con 0x7f2f4c0a9d70 2024-03-29T10:44:38.857 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:38.872+0000 7f2f497fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:44:38.857 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:38.872+0000 7f2f497fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:44:38.858 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:38.875+0000 7f2f53225640 10 librados: watch_flush enter 2024-03-29T10:44:38.861 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:38.879+0000 7f2f53225640 10 librados: watch_flush exit 2024-03-29T10:44:38.861 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:38.879+0000 7f2f53225640 20 client.4357.objecter shutdown clearing up homeless session... 2024-03-29T10:44:38.861 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:38.879+0000 7f2f53225640 10 client.4357.objecter successfully canceled tick 2024-03-29T10:44:38.862 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:38.879+0000 7f2f53225640 1 -- 172.21.15.88:0/2776229174 >> v1:172.21.15.88:6816/1752861349 conn(0x7f2f3806def0 legacy=0x7f2f380703b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:38.862 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:38.879+0000 7f2f53225640 10 monclient: shutdown 2024-03-29T10:44:38.862 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:38.879+0000 7f2f53225640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:44:38.862 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:38.879+0000 7f2f53225640 1 -- 172.21.15.88:0/2776229174 >> v1:172.21.15.196:6789/0 conn(0x7f2f4c0a9d70 legacy=0x7f2f4c0b0df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:38.862 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:38.879+0000 7f2f53225640 1 -- 172.21.15.88:0/2776229174 shutdown_connections 2024-03-29T10:44:38.863 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:38.879+0000 7f2f53225640 1 -- 172.21.15.88:0/2776229174 >> 172.21.15.88:0/2776229174 conn(0x7f2f4c066a00 msgr2=0x7f2f4c066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:44:38.863 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:38.879+0000 7f2f53225640 1 -- 172.21.15.88:0/2776229174 shutdown_connections 2024-03-29T10:44:38.863 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:38.879+0000 7f2f53225640 1 -- 172.21.15.88:0/2776229174 wait complete. 2024-03-29T10:44:38.863 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:38.879+0000 7f2f53225640 1 librados: shutdown 2024-03-29T10:44:38.877 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:44:45.685 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:44:45.844 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:45.861+0000 7fbfeee35640 10 monclient: get_monmap_and_config 2024-03-29T10:44:45.845 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:45.861+0000 7fbfeee35640 10 monclient: build_initial_monmap 2024-03-29T10:44:45.845 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:45.861+0000 7fbfeee35640 10 monclient: monmap: 2024-03-29T10:44:45.845 INFO:teuthology.misc.health.smithi088.stderr:epoch 0 2024-03-29T10:44:45.845 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:44:45.845 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:44:45.862972+0000 2024-03-29T10:44:45.845 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:44:45.862972+0000 2024-03-29T10:44:45.845 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 0 (unknown) 2024-03-29T10:44:45.845 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:44:45.845 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-03-29T10:44:45.845 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-29T10:44:45.845 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-b 2024-03-29T10:44:45.845 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:44:45.846 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:45.863+0000 7fbfeee35640 1 Processor -- start 2024-03-29T10:44:45.846 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:45.863+0000 7fbfeee35640 1 -- start start 2024-03-29T10:44:45.846 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:45.863+0000 7fbfeee35640 10 monclient: init 2024-03-29T10:44:45.849 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:45.864+0000 7fbfeee35640 10 monclient: _reopen_session rank -1 2024-03-29T10:44:45.849 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:45.864+0000 7fbfeee35640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:44:45.849 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:45.864+0000 7fbfeee35640 10 monclient(hunting): picked mon.noname-b con 0x7fbfe81422b0 addr v1:172.21.15.88:6790/0 2024-03-29T10:44:45.849 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:45.864+0000 7fbfeee35640 10 monclient(hunting): picked mon.noname-c con 0x7fbfe8142690 addr v1:172.21.15.196:6789/0 2024-03-29T10:44:45.849 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:45.864+0000 7fbfeee35640 10 monclient(hunting): picked mon.noname-a con 0x7fbfe80b17d0 addr v1:172.21.15.88:6789/0 2024-03-29T10:44:45.849 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:45.864+0000 7fbfeee35640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbfe8142c80 con 0x7fbfe80b17d0 2024-03-29T10:44:45.849 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:45.864+0000 7fbfeee35640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbfe80aae40 con 0x7fbfe8142690 2024-03-29T10:44:45.849 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:45.864+0000 7fbfeee35640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbfe813fbd0 con 0x7fbfe81422b0 2024-03-29T10:44:45.849 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:45.864+0000 7fbfeee35640 10 monclient(hunting): _renew_subs 2024-03-29T10:44:45.849 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:45.864+0000 7fbfeee35640 10 monclient(hunting): authenticate will time out at 1008.100322s 2024-03-29T10:44:45.850 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:45.867+0000 7fbfecbd1640 1 --1- >> v1:172.21.15.88:6790/0 conn(0x7fbfe81422b0 0x7fbfe813f4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6790/0 says I am v1:172.21.15.88:54307/0 (socket says 172.21.15.88:54307) 2024-03-29T10:44:45.850 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:45.867+0000 7fbfecbd1640 1 -- 172.21.15.88:0/2692643887 learned_addr learned my addr 172.21.15.88:0/2692643887 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-29T10:44:45.852 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:45.869+0000 7fbfe77fe640 1 -- 172.21.15.88:0/2692643887 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1858587810 0 0) 0x7fbfe813fbd0 con 0x7fbfe81422b0 2024-03-29T10:44:45.852 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:45.869+0000 7fbfe77fe640 10 monclient(hunting): _negotiate 2024-03-29T10:44:45.852 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:45.869+0000 7fbfe77fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:45.852 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:45.869+0000 7fbfe77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:45.852 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:45.869+0000 7fbfe77fe640 10 monclient(hunting): my global_id is 4388 2024-03-29T10:44:45.852 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:45.869+0000 7fbfe77fe640 1 -- 172.21.15.88:0/2692643887 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbfd0003520 con 0x7fbfe81422b0 2024-03-29T10:44:45.853 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:45.870+0000 7fbfe77fe640 1 -- 172.21.15.88:0/2692643887 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2630429587 0 0) 0x7fbfd0003520 con 0x7fbfe81422b0 2024-03-29T10:44:45.853 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:45.870+0000 7fbfe77fe640 1 -- 172.21.15.88:0/2692643887 >> v1:172.21.15.196:6789/0 conn(0x7fbfe8142690 legacy=0x7fbfe80b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:45.853 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:45.870+0000 7fbfe77fe640 1 -- 172.21.15.88:0/2692643887 >> v1:172.21.15.88:6789/0 conn(0x7fbfe80b17d0 legacy=0x7fbfe80b2ca0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:44:45.856 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:45.871+0000 7fbfe77fe640 10 monclient: _finish_hunting 0 2024-03-29T10:44:45.856 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:45.871+0000 7fbfe77fe640 1 monclient: found mon.noname-b 2024-03-29T10:44:45.856 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:45.871+0000 7fbfe77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:44:45.856 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:45.871+0000 7fbfe77fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.88:6790/0 2024-03-29T10:44:45.856 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:45.871+0000 7fbfe77fe640 1 -- 172.21.15.88:0/2692643887 --> v1:172.21.15.88:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfe8151940 con 0x7fbfe81422b0 2024-03-29T10:44:45.856 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:45.871+0000 7fbfe77fe640 10 monclient: _finish_auth 0 2024-03-29T10:44:45.856 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:45.871+0000 7fbfe77fe640 10 monclient: _check_auth_tickets 2024-03-29T10:44:45.857 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:45.871+0000 7fbfe77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:44:45.857 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:45.871+0000 7fbfeee35640 5 monclient: authenticate success, global_id 4388 2024-03-29T10:44:45.857 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:45.871+0000 7fbfeee35640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:44:45.857 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:45.871+0000 7fbfe77fe640 1 -- 172.21.15.88:0/2692643887 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7fbfe0001a10 con 0x7fbfe81422b0 2024-03-29T10:44:45.857 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:45.871+0000 7fbfe77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:45.857 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:45.871+0000 7fbfe77fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:44:45.857 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:45.871+0000 7fbfe77fe640 10 monclient: dump: 2024-03-29T10:44:45.857 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:44:45.857 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:44:45.857 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:44:45.857 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:44:45.857 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:45.857 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:44:45.857 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:44:45.858 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:44:45.858 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:44:45.858 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:44:45.858 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:45.872+0000 7fbfeee35640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:44:45.858 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:45.872+0000 7fbfe77fe640 1 -- 172.21.15.88:0/2692643887 <== mon.2 v1:172.21.15.88:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbfe0001c30 con 0x7fbfe81422b0 2024-03-29T10:44:45.858 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:45.872+0000 7fbfe77fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:44:45.858 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:45.872+0000 7fbfeee35640 10 monclient: get_monmap_and_config success 2024-03-29T10:44:45.858 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:45.872+0000 7fbfeee35640 4 set_mon_vals no callback set 2024-03-29T10:44:45.858 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:45.872+0000 7fbfeee35640 10 monclient: shutdown 2024-03-29T10:44:45.858 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:45.872+0000 7fbfeee35640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:44:45.858 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:45.872+0000 7fbfeee35640 1 -- 172.21.15.88:0/2692643887 >> v1:172.21.15.88:6790/0 conn(0x7fbfe81422b0 legacy=0x7fbfe813f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:45.859 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:45.876+0000 7fbfeee35640 1 -- 172.21.15.88:0/2692643887 shutdown_connections 2024-03-29T10:44:45.859 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:45.876+0000 7fbfeee35640 1 -- 172.21.15.88:0/2692643887 >> 172.21.15.88:0/2692643887 conn(0x7fbfe8066a00 msgr2=0x7fbfe8066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:44:45.862 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:45.878+0000 7fbfeee35640 1 -- 172.21.15.88:0/2692643887 shutdown_connections 2024-03-29T10:44:45.862 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:45.878+0000 7fbfeee35640 1 -- 172.21.15.88:0/2692643887 wait complete. 2024-03-29T10:44:45.862 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:45.879+0000 7fbfeee35640 10 monclient: build_initial_monmap 2024-03-29T10:44:45.863 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:45.879+0000 7fbfeee35640 10 monclient: monmap: 2024-03-29T10:44:45.863 INFO:teuthology.misc.health.smithi088.stderr:epoch 0 2024-03-29T10:44:45.863 INFO:teuthology.misc.health.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:44:45.863 INFO:teuthology.misc.health.smithi088.stderr:last_changed 0.000000 2024-03-29T10:44:45.863 INFO:teuthology.misc.health.smithi088.stderr:created 0.000000 2024-03-29T10:44:45.863 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 0 (unknown) 2024-03-29T10:44:45.863 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:44:45.863 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-03-29T10:44:45.863 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-29T10:44:45.863 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-c 2024-03-29T10:44:45.863 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:44:45.863 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:45.879+0000 7fbfeee35640 1 librados: starting msgr at 2024-03-29T10:44:45.863 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:45.879+0000 7fbfeee35640 1 librados: starting objecter 2024-03-29T10:44:45.863 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:45.880+0000 7fbfeee35640 1 Processor -- start 2024-03-29T10:44:45.864 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:45.880+0000 7fbfeee35640 1 -- start start 2024-03-29T10:44:45.864 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:45.880+0000 7fbfeee35640 1 librados: setting wanted keys 2024-03-29T10:44:45.864 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:45.880+0000 7fbfeee35640 1 librados: calling monclient init 2024-03-29T10:44:45.864 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:45.880+0000 7fbfeee35640 10 monclient: init 2024-03-29T10:44:45.866 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:45.881+0000 7fbfeee35640 10 monclient: _reopen_session rank -1 2024-03-29T10:44:45.866 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:45.881+0000 7fbfeee35640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:44:45.866 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:45.881+0000 7fbfeee35640 10 monclient(hunting): picked mon.noname-a con 0x7fbfe80b17d0 addr v1:172.21.15.88:6789/0 2024-03-29T10:44:45.866 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:45.881+0000 7fbfeee35640 10 monclient(hunting): picked mon.noname-b con 0x7fbfe81422b0 addr v1:172.21.15.196:6789/0 2024-03-29T10:44:45.866 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:45.881+0000 7fbfeee35640 10 monclient(hunting): picked mon.noname-c con 0x7fbfe8142690 addr v1:172.21.15.88:6790/0 2024-03-29T10:44:45.866 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:45.881+0000 7fbfeee35640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbfe80b76d0 con 0x7fbfe80b17d0 2024-03-29T10:44:45.866 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:45.881+0000 7fbfeee35640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbfe812d7f0 con 0x7fbfe81422b0 2024-03-29T10:44:45.866 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:45.881+0000 7fbfeee35640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbfe80ac9f0 con 0x7fbfe8142690 2024-03-29T10:44:45.866 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:45.881+0000 7fbfeee35640 10 monclient(hunting): _renew_subs 2024-03-29T10:44:45.866 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:45.881+0000 7fbfeee35640 10 monclient(hunting): authenticate will time out at 1008.117005s 2024-03-29T10:44:45.867 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:45.885+0000 7fbfecbd1640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fbfe80b17d0 0x7fbfe80b6cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:37371/0 (socket says 172.21.15.88:37371) 2024-03-29T10:44:45.867 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:45.885+0000 7fbfecbd1640 1 -- 172.21.15.88:0/2608434864 learned_addr learned my addr 172.21.15.88:0/2608434864 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-29T10:44:45.868 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:45.885+0000 7fbfe57fa640 10 client.?.objecter ms_handle_connect 0x7fbfe80b17d0 2024-03-29T10:44:45.868 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:45.885+0000 7fbfe57fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:44:45.870 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:45.887+0000 7fbfe57fa640 10 client.?.objecter ms_handle_connect 0x7fbfe8142690 2024-03-29T10:44:45.870 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:45.887+0000 7fbfe57fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:44:45.870 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:45.887+0000 7fbfe57fa640 1 -- 172.21.15.88:0/2608434864 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3481796823 0 0) 0x7fbfe80ac9f0 con 0x7fbfe8142690 2024-03-29T10:44:45.870 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:45.887+0000 7fbfe57fa640 10 monclient(hunting): _negotiate 2024-03-29T10:44:45.870 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:45.887+0000 7fbfe57fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:45.870 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:45.887+0000 7fbfe57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:45.870 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:45.887+0000 7fbfe57fa640 10 monclient(hunting): my global_id is 4391 2024-03-29T10:44:45.870 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:45.887+0000 7fbfe57fa640 1 -- 172.21.15.88:0/2608434864 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbfd4004eb0 con 0x7fbfe8142690 2024-03-29T10:44:45.875 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:45.888+0000 7fbfe57fa640 10 client.?.objecter ms_handle_connect 0x7fbfe81422b0 2024-03-29T10:44:45.875 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:45.888+0000 7fbfe57fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:44:45.875 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:45.888+0000 7fbfe57fa640 1 -- 172.21.15.88:0/2608434864 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2737438547 0 0) 0x7fbfe80b76d0 con 0x7fbfe80b17d0 2024-03-29T10:44:45.875 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:45.888+0000 7fbfe57fa640 10 monclient(hunting): _negotiate 2024-03-29T10:44:45.876 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:45.888+0000 7fbfe57fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:45.876 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:45.888+0000 7fbfe57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:45.876 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:45.888+0000 7fbfe57fa640 10 monclient(hunting): my global_id is 4362 2024-03-29T10:44:45.876 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:45.888+0000 7fbfe57fa640 1 -- 172.21.15.88:0/2608434864 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbfe80ac9f0 con 0x7fbfe80b17d0 2024-03-29T10:44:45.876 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:45.889+0000 7fbfe57fa640 1 -- 172.21.15.88:0/2608434864 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4273607493 0 0) 0x7fbfd4004eb0 con 0x7fbfe8142690 2024-03-29T10:44:45.876 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:45.889+0000 7fbfe57fa640 1 -- 172.21.15.88:0/2608434864 --> v1:172.21.15.88:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbfe80b76d0 con 0x7fbfe8142690 2024-03-29T10:44:45.876 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:45.889+0000 7fbfe57fa640 1 -- 172.21.15.88:0/2608434864 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7fbfd8001770 con 0x7fbfe8142690 2024-03-29T10:44:45.876 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:45.889+0000 7fbfe57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:45.876 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:45.889+0000 7fbfe57fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:44:45.876 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:45.889+0000 7fbfe57fa640 10 monclient(hunting): dump: 2024-03-29T10:44:45.876 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:44:45.876 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:44:45.876 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:44:45.877 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:44:45.877 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:45.877 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:44:45.877 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:44:45.877 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:44:45.877 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:44:45.877 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:44:45.877 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:45.889+0000 7fbfe57fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:44:45.877 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:45.889+0000 7fbfe57fa640 1 -- 172.21.15.88:0/2608434864 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4150490031 0 0) 0x7fbfe812d7f0 con 0x7fbfe81422b0 2024-03-29T10:44:45.877 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:45.889+0000 7fbfe57fa640 10 monclient(hunting): _negotiate 2024-03-29T10:44:45.877 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:45.889+0000 7fbfe57fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:45.877 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:45.889+0000 7fbfe57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:45.877 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:45.889+0000 7fbfe57fa640 10 monclient(hunting): my global_id is 4363 2024-03-29T10:44:45.878 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:45.889+0000 7fbfe57fa640 1 -- 172.21.15.88:0/2608434864 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbfd4004eb0 con 0x7fbfe81422b0 2024-03-29T10:44:45.878 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:45.889+0000 7fbfe57fa640 1 -- 172.21.15.88:0/2608434864 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1233960487 0 0) 0x7fbfe80ac9f0 con 0x7fbfe80b17d0 2024-03-29T10:44:45.878 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:45.889+0000 7fbfe57fa640 1 -- 172.21.15.88:0/2608434864 --> v1:172.21.15.88:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbfe812d7f0 con 0x7fbfe80b17d0 2024-03-29T10:44:45.878 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:45.889+0000 7fbfe57fa640 1 -- 172.21.15.88:0/2608434864 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7fbfe0004d00 con 0x7fbfe80b17d0 2024-03-29T10:44:45.878 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:45.889+0000 7fbfe57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:45.878 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:45.889+0000 7fbfe57fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:44:45.878 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:45.889+0000 7fbfe57fa640 10 monclient(hunting): dump: 2024-03-29T10:44:45.878 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:44:45.878 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:44:45.878 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:44:45.878 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:44:45.879 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:45.879 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:44:45.879 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:44:45.879 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:44:45.879 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:44:45.879 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:44:45.879 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:45.889+0000 7fbfe57fa640 1 -- 172.21.15.88:0/2608434864 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1577872159 0 0) 0x7fbfd4004eb0 con 0x7fbfe81422b0 2024-03-29T10:44:45.879 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:45.889+0000 7fbfe57fa640 1 -- 172.21.15.88:0/2608434864 --> v1:172.21.15.196:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbfe80ac9f0 con 0x7fbfe81422b0 2024-03-29T10:44:45.879 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:45.889+0000 7fbfe57fa640 1 -- 172.21.15.88:0/2608434864 <== mon.0 v1:172.21.15.88:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 73859467 0 0) 0x7fbfe812d7f0 con 0x7fbfe80b17d0 2024-03-29T10:44:45.879 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:45.889+0000 7fbfe57fa640 1 -- 172.21.15.88:0/2608434864 >> v1:172.21.15.88:6790/0 conn(0x7fbfe8142690 legacy=0x7fbfe812d0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:45.879 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:45.889+0000 7fbfe57fa640 1 -- 172.21.15.88:0/2608434864 >> v1:172.21.15.196:6789/0 conn(0x7fbfe81422b0 legacy=0x7fbfe812c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:45.879 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:45.889+0000 7fbfe57fa640 10 monclient: _finish_hunting 0 2024-03-29T10:44:45.879 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:45.889+0000 7fbfe57fa640 1 monclient: found mon.a 2024-03-29T10:44:45.879 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:45.889+0000 7fbfe57fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:44:45.880 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:45.889+0000 7fbfe57fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-03-29T10:44:45.880 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:45.889+0000 7fbfe57fa640 1 -- 172.21.15.88:0/2608434864 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfe80accd0 con 0x7fbfe80b17d0 2024-03-29T10:44:45.880 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:45.890+0000 7fbfe57fa640 10 monclient: _finish_auth 0 2024-03-29T10:44:45.880 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:45.890+0000 7fbfe57fa640 10 monclient: _check_auth_tickets 2024-03-29T10:44:45.880 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:45.890+0000 7fbfe57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:44:45.880 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:45.890+0000 7fbfe57fa640 1 -- 172.21.15.88:0/2608434864 <== mon.0 v1:172.21.15.88:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbfe00051d0 con 0x7fbfe80b17d0 2024-03-29T10:44:45.880 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:45.890+0000 7fbfe57fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:44:45.880 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:45.890+0000 7fbfe57fa640 1 -- 172.21.15.88:0/2608434864 <== mon.0 v1:172.21.15.88:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7fbfe00055a0 con 0x7fbfe80b17d0 2024-03-29T10:44:45.880 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:45.890+0000 7fbfe57fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:45.880 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:45.890+0000 7fbfe57fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:44:45.880 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:45.890+0000 7fbfe57fa640 10 monclient: dump: 2024-03-29T10:44:45.880 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:44:45.880 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:44:45.880 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:44:45.881 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:44:45.881 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:45.881 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:44:45.881 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:44:45.881 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:44:45.881 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:44:45.881 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:44:45.881 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:45.890+0000 7fbfeee35640 5 monclient: authenticate success, global_id 4362 2024-03-29T10:44:45.881 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:45.890+0000 7fbfeee35640 10 monclient: _renew_subs 2024-03-29T10:44:45.881 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:45.890+0000 7fbfeee35640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-03-29T10:44:45.881 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:45.890+0000 7fbfeee35640 1 -- 172.21.15.88:0/2608434864 --> v1:172.21.15.88:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbfe812dba0 con 0x7fbfe80b17d0 2024-03-29T10:44:45.881 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:45.890+0000 7fbfeee35640 10 client.4362.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:44:45.881 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:45.890+0000 7fbfeee35640 10 monclient: _renew_subs 2024-03-29T10:44:45.881 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:45.890+0000 7fbfeee35640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-03-29T10:44:45.881 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:45.890+0000 7fbfeee35640 1 -- 172.21.15.88:0/2608434864 --> v1:172.21.15.88:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbfe8150540 con 0x7fbfe80b17d0 2024-03-29T10:44:45.882 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:45.890+0000 7fbfeee35640 1 librados: init done 2024-03-29T10:44:45.882 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:45.890+0000 7fbfe67fc640 4 set_mon_vals no callback set 2024-03-29T10:44:45.882 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:45.892+0000 7fbfe57fa640 1 -- 172.21.15.88:0/2608434864 <== mon.0 v1:172.21.15.88:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4235629220 0 0) 0x7fbfe0001e20 con 0x7fbfe80b17d0 2024-03-29T10:44:45.883 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:45.893+0000 7fbfeee35640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:44:45.883 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:45.893+0000 7fbfeee35640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:44:45.883 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:45.893+0000 7fbfeee35640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-03-29T10:44:45.883 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:45.893+0000 7fbfeee35640 1 -- 172.21.15.88:0/2608434864 --> v1:172.21.15.88:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbfc8005080 con 0x7fbfe80b17d0 2024-03-29T10:44:45.883 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:45.893+0000 7fbfe57fa640 1 -- 172.21.15.88:0/2608434864 <== mon.0 v1:172.21.15.88:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4257673582 0 0) 0x7fbfe0084fb0 con 0x7fbfe80b17d0 2024-03-29T10:44:45.883 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:45.893+0000 7fbfe57fa640 10 client.4362.objecter ms_dispatch 0x7fbfe8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:44:45.883 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:45.893+0000 7fbfe57fa640 3 client.4362.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:44:45.884 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:45.893+0000 7fbfe57fa640 3 client.4362.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:44:45.884 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:45.893+0000 7fbfe57fa640 20 client.4362.objecter dump_active .. 0 homeless 2024-03-29T10:44:45.884 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:45.893+0000 7fbfe7fff640 10 monclient: get_auth_request con 0x7fbfd406e070 auth_method 0 2024-03-29T10:44:45.884 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:45.894+0000 7fbfe57fa640 10 client.4362.objecter ms_handle_connect 0x7fbfd406e070 2024-03-29T10:44:45.887 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:45.904+0000 7fbfe57fa640 1 -- 172.21.15.88:0/2608434864 <== mon.0 v1:172.21.15.88:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fbfe0050890 con 0x7fbfe80b17d0 2024-03-29T10:44:45.887 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:45.904+0000 7fbfe57fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:44:45.887 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:45.904+0000 7fbfe57fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:44:46.210 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:46.227+0000 7fbfeee35640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:44:46.210 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:46.227+0000 7fbfeee35640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:44:46.210 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:46.227+0000 7fbfeee35640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-03-29T10:44:46.210 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:46.227+0000 7fbfeee35640 1 -- 172.21.15.88:0/2608434864 --> v1:172.21.15.88:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbfc8005330 con 0x7fbfe80b17d0 2024-03-29T10:44:46.213 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:46.230+0000 7fbfe57fa640 1 -- 172.21.15.88:0/2608434864 <== mon.0 v1:172.21.15.88:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 344506374) 0x7fbfe00527c0 con 0x7fbfe80b17d0 2024-03-29T10:44:46.213 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:46.230+0000 7fbfe57fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:44:46.213 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:46.230+0000 7fbfe57fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:44:46.214 INFO:teuthology.misc.health.smithi088.stdout: 2024-03-29T10:44:46.214 INFO:teuthology.misc.health.smithi088.stdout:{"epoch":10,"fsid":"869f5188-f92b-42dd-a46d-8c646e0388a9","created":"2024-03-29T10:40:40.285087+0000","modified":"2024-03-29T10:40:42.536230+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:40:42.536230+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a26c0c1c-8de0-438c-a3f6-3df10ea61574","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5de24915-8dec-49b6-8b16-f3170197c591","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ac054af3-d589-4565-9644-e6af75975c53","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3ec4f045-c714-4a22-bd1a-e07330b45ab3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"cddb9867-8e30-4951-9b13-3d1e78b9c45a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dc6d9911-785d-432f-8b53-56a1ccab302f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"77c88437-b587-496e-8b4d-1824ea8b5305","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ce963d96-5a8e-44c3-99de-c0dbebe234ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:44:46.218 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:46.234+0000 7fbfeee35640 10 librados: watch_flush enter 2024-03-29T10:44:46.218 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:46.235+0000 7fbfeee35640 10 librados: watch_flush exit 2024-03-29T10:44:46.218 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:46.235+0000 7fbfeee35640 20 client.4362.objecter shutdown clearing up homeless session... 2024-03-29T10:44:46.218 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:46.235+0000 7fbfeee35640 10 client.4362.objecter successfully canceled tick 2024-03-29T10:44:46.219 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:46.235+0000 7fbfeee35640 1 -- 172.21.15.88:0/2608434864 >> v1:172.21.15.88:6816/1752861349 conn(0x7fbfd406e070 legacy=0x7fbfd4070530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:46.219 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:46.236+0000 7fbfeee35640 10 monclient: shutdown 2024-03-29T10:44:46.219 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:46.236+0000 7fbfeee35640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:44:46.219 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:46.236+0000 7fbfeee35640 1 -- 172.21.15.88:0/2608434864 >> v1:172.21.15.88:6789/0 conn(0x7fbfe80b17d0 legacy=0x7fbfe80b6cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:46.219 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:46.236+0000 7fbfeee35640 1 -- 172.21.15.88:0/2608434864 shutdown_connections 2024-03-29T10:44:46.219 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:46.236+0000 7fbfeee35640 1 -- 172.21.15.88:0/2608434864 >> 172.21.15.88:0/2608434864 conn(0x7fbfe8066a00 msgr2=0x7fbfe812f970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:44:46.220 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:46.236+0000 7fbfeee35640 1 -- 172.21.15.88:0/2608434864 shutdown_connections 2024-03-29T10:44:46.220 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:46.236+0000 7fbfeee35640 1 -- 172.21.15.88:0/2608434864 wait complete. 2024-03-29T10:44:46.220 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:46.236+0000 7fbfeee35640 1 librados: shutdown 2024-03-29T10:44:46.237 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:44:53.044 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:44:53.211 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:53.228+0000 7f3c071e8640 10 monclient: get_monmap_and_config 2024-03-29T10:44:53.211 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:53.228+0000 7f3c071e8640 10 monclient: build_initial_monmap 2024-03-29T10:44:53.211 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:53.228+0000 7f3c071e8640 10 monclient: monmap: 2024-03-29T10:44:53.212 INFO:teuthology.misc.health.smithi088.stderr:epoch 0 2024-03-29T10:44:53.212 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:44:53.212 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:44:53.229386+0000 2024-03-29T10:44:53.212 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:44:53.229386+0000 2024-03-29T10:44:53.212 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 0 (unknown) 2024-03-29T10:44:53.212 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:44:53.212 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-03-29T10:44:53.212 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-29T10:44:53.212 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-b 2024-03-29T10:44:53.212 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:44:53.213 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:53.229+0000 7f3c071e8640 1 Processor -- start 2024-03-29T10:44:53.213 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:53.230+0000 7f3c071e8640 1 -- start start 2024-03-29T10:44:53.213 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:53.230+0000 7f3c071e8640 10 monclient: init 2024-03-29T10:44:53.214 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:53.231+0000 7f3c071e8640 10 monclient: _reopen_session rank -1 2024-03-29T10:44:53.214 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:53.231+0000 7f3c071e8640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:44:53.214 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:53.231+0000 7f3c071e8640 10 monclient(hunting): picked mon.noname-b con 0x7f3bf8006ad0 addr v1:172.21.15.88:6790/0 2024-03-29T10:44:53.214 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:53.231+0000 7f3c071e8640 10 monclient(hunting): picked mon.noname-c con 0x7f3bf80c5120 addr v1:172.21.15.196:6789/0 2024-03-29T10:44:53.214 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:53.231+0000 7f3c071e8640 10 monclient(hunting): picked mon.noname-a con 0x7f3bf80cb780 addr v1:172.21.15.88:6789/0 2024-03-29T10:44:53.214 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:53.231+0000 7f3c071e8640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3bf8008ed0 con 0x7f3bf80cb780 2024-03-29T10:44:53.214 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:53.231+0000 7f3c071e8640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3bf802ffb0 con 0x7f3bf80c5120 2024-03-29T10:44:53.214 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:53.231+0000 7f3c071e8640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3bf80cbc10 con 0x7f3bf8006ad0 2024-03-29T10:44:53.214 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:53.231+0000 7f3c071e8640 10 monclient(hunting): _renew_subs 2024-03-29T10:44:53.214 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:53.231+0000 7f3c071e8640 10 monclient(hunting): authenticate will time out at 1015.467401s 2024-03-29T10:44:53.221 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:53.238+0000 7f3bfffff640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f3bf80c5120 0x7f3bf80cb070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.88:39583/0 (socket says 172.21.15.88:39583) 2024-03-29T10:44:53.221 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:53.238+0000 7f3bfffff640 1 -- 172.21.15.88:0/3377905483 learned_addr learned my addr 172.21.15.88:0/3377905483 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-29T10:44:53.226 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:53.239+0000 7f3bff7fe640 1 -- 172.21.15.88:0/3377905483 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3600926725 0 0) 0x7f3bf802ffb0 con 0x7f3bf80c5120 2024-03-29T10:44:53.226 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:53.239+0000 7f3bff7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:44:53.226 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:53.239+0000 7f3bff7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:53.226 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:53.239+0000 7f3bff7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:53.226 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:53.239+0000 7f3bff7fe640 10 monclient(hunting): my global_id is 4366 2024-03-29T10:44:53.226 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:53.239+0000 7f3bff7fe640 1 -- 172.21.15.88:0/3377905483 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3bf0003520 con 0x7f3bf80c5120 2024-03-29T10:44:53.226 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:53.239+0000 7f3bff7fe640 1 -- 172.21.15.88:0/3377905483 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1205691387 0 0) 0x7f3bf8008ed0 con 0x7f3bf80cb780 2024-03-29T10:44:53.227 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:53.239+0000 7f3bff7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:44:53.227 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:53.239+0000 7f3bff7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:53.227 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:53.239+0000 7f3bff7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:53.227 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:53.239+0000 7f3bff7fe640 10 monclient(hunting): my global_id is 4365 2024-03-29T10:44:53.227 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:53.239+0000 7f3bff7fe640 1 -- 172.21.15.88:0/3377905483 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3bf802ffb0 con 0x7f3bf80cb780 2024-03-29T10:44:53.227 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:53.239+0000 7f3bff7fe640 1 -- 172.21.15.88:0/3377905483 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1766953535 0 0) 0x7f3bf80cbc10 con 0x7f3bf8006ad0 2024-03-29T10:44:53.227 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:53.239+0000 7f3bff7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:44:53.227 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:53.239+0000 7f3bff7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:53.227 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:53.239+0000 7f3bff7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:53.227 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:53.239+0000 7f3bff7fe640 10 monclient(hunting): my global_id is 4394 2024-03-29T10:44:53.227 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:53.239+0000 7f3bff7fe640 1 -- 172.21.15.88:0/3377905483 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3bf8008ed0 con 0x7f3bf8006ad0 2024-03-29T10:44:53.227 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:53.239+0000 7f3bff7fe640 1 -- 172.21.15.88:0/3377905483 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2284935293 0 0) 0x7f3bf802ffb0 con 0x7f3bf80cb780 2024-03-29T10:44:53.227 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:53.239+0000 7f3bff7fe640 1 -- 172.21.15.88:0/3377905483 >> v1:172.21.15.88:6790/0 conn(0x7f3bf8006ad0 legacy=0x7f3bf80c4a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:53.228 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:53.239+0000 7f3bff7fe640 1 -- 172.21.15.88:0/3377905483 >> v1:172.21.15.196:6789/0 conn(0x7f3bf80c5120 legacy=0x7f3bf80cb070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:53.228 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:53.239+0000 7f3bff7fe640 10 monclient: _finish_hunting 0 2024-03-29T10:44:53.228 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:53.239+0000 7f3bff7fe640 1 monclient: found mon.noname-a 2024-03-29T10:44:53.228 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:53.239+0000 7f3bff7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:44:53.228 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:53.239+0000 7f3bff7fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.88:6789/0 2024-03-29T10:44:53.228 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:53.239+0000 7f3bff7fe640 1 -- 172.21.15.88:0/3377905483 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bf8030220 con 0x7f3bf80cb780 2024-03-29T10:44:53.228 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:53.239+0000 7f3bff7fe640 10 monclient: _finish_auth 0 2024-03-29T10:44:53.228 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:53.239+0000 7f3bff7fe640 10 monclient: _check_auth_tickets 2024-03-29T10:44:53.228 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:53.239+0000 7f3bff7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:44:53.228 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:53.239+0000 7f3bff7fe640 1 -- 172.21.15.88:0/3377905483 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f3c0004f730 con 0x7f3bf80cb780 2024-03-29T10:44:53.228 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:53.239+0000 7f3bff7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:53.228 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:53.239+0000 7f3bff7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:44:53.228 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:53.239+0000 7f3bff7fe640 10 monclient: dump: 2024-03-29T10:44:53.228 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:44:53.228 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:44:53.229 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:44:53.229 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:44:53.229 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:53.229 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:44:53.229 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:44:53.229 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:44:53.229 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:44:53.229 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:44:53.229 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:53.239+0000 7f3bff7fe640 1 -- 172.21.15.88:0/3377905483 <== mon.0 v1:172.21.15.88:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3c00075f10 con 0x7f3bf80cb780 2024-03-29T10:44:53.229 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:53.239+0000 7f3bff7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:44:53.229 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:53.239+0000 7f3bff7fe640 1 -- 172.21.15.88:0/3377905483 <== mon.0 v1:172.21.15.88:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f3c000758b0 con 0x7f3bf80cb780 2024-03-29T10:44:53.229 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:53.239+0000 7f3bff7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:53.230 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:53.239+0000 7f3bff7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:44:53.230 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:53.239+0000 7f3bff7fe640 10 monclient: dump: 2024-03-29T10:44:53.230 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:44:53.230 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:44:53.230 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:44:53.230 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:44:53.230 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:53.230 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:44:53.230 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:44:53.230 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:44:53.230 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:44:53.231 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:44:53.231 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:53.240+0000 7f3c071e8640 5 monclient: authenticate success, global_id 4365 2024-03-29T10:44:53.231 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:53.240+0000 7f3c071e8640 10 monclient: get_monmap_and_config success 2024-03-29T10:44:53.231 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:53.240+0000 7f3c071e8640 4 set_mon_vals no callback set 2024-03-29T10:44:53.231 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:53.241+0000 7f3c071e8640 10 monclient: shutdown 2024-03-29T10:44:53.231 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:53.241+0000 7f3c071e8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:44:53.231 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:53.241+0000 7f3c071e8640 1 -- 172.21.15.88:0/3377905483 >> v1:172.21.15.88:6789/0 conn(0x7f3bf80cb780 legacy=0x7f3bf80d3130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:53.232 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:53.243+0000 7f3c071e8640 1 -- 172.21.15.88:0/3377905483 shutdown_connections 2024-03-29T10:44:53.232 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:53.243+0000 7f3c071e8640 1 -- 172.21.15.88:0/3377905483 >> 172.21.15.88:0/3377905483 conn(0x7f3bf80058f0 msgr2=0x7f3bf8005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:44:53.232 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:53.244+0000 7f3c071e8640 1 -- 172.21.15.88:0/3377905483 shutdown_connections 2024-03-29T10:44:53.233 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:53.244+0000 7f3c071e8640 1 -- 172.21.15.88:0/3377905483 wait complete. 2024-03-29T10:44:53.233 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:53.246+0000 7f3c071e8640 10 monclient: build_initial_monmap 2024-03-29T10:44:53.233 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:53.246+0000 7f3c071e8640 10 monclient: monmap: 2024-03-29T10:44:53.233 INFO:teuthology.misc.health.smithi088.stderr:epoch 0 2024-03-29T10:44:53.233 INFO:teuthology.misc.health.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:44:53.233 INFO:teuthology.misc.health.smithi088.stderr:last_changed 0.000000 2024-03-29T10:44:53.233 INFO:teuthology.misc.health.smithi088.stderr:created 0.000000 2024-03-29T10:44:53.233 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 0 (unknown) 2024-03-29T10:44:53.233 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:44:53.233 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-03-29T10:44:53.233 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-29T10:44:53.234 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-c 2024-03-29T10:44:53.234 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:44:53.234 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:53.246+0000 7f3c071e8640 1 librados: starting msgr at 2024-03-29T10:44:53.234 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:53.249+0000 7f3c071e8640 1 librados: starting objecter 2024-03-29T10:44:53.234 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:53.249+0000 7f3c071e8640 1 Processor -- start 2024-03-29T10:44:53.235 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:53.251+0000 7f3c071e8640 1 -- start start 2024-03-29T10:44:53.235 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:53.251+0000 7f3c071e8640 1 librados: setting wanted keys 2024-03-29T10:44:53.235 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:53.251+0000 7f3c071e8640 1 librados: calling monclient init 2024-03-29T10:44:53.235 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:53.251+0000 7f3c071e8640 10 monclient: init 2024-03-29T10:44:53.235 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:53.252+0000 7f3c071e8640 10 monclient: _reopen_session rank -1 2024-03-29T10:44:53.236 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:53.252+0000 7f3c071e8640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:44:53.236 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:53.252+0000 7f3c071e8640 10 monclient(hunting): picked mon.noname-c con 0x7f3bf8006ad0 addr v1:172.21.15.88:6790/0 2024-03-29T10:44:53.236 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:53.252+0000 7f3c071e8640 10 monclient(hunting): picked mon.noname-b con 0x7f3bf80c5120 addr v1:172.21.15.196:6789/0 2024-03-29T10:44:53.236 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:53.252+0000 7f3c071e8640 10 monclient(hunting): picked mon.noname-a con 0x7f3bf80cb780 addr v1:172.21.15.88:6789/0 2024-03-29T10:44:53.236 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:53.252+0000 7f3c071e8640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3bf815c4f0 con 0x7f3bf80cb780 2024-03-29T10:44:53.236 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:53.252+0000 7f3c071e8640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3bf80cab10 con 0x7f3bf80c5120 2024-03-29T10:44:53.236 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:53.252+0000 7f3c071e8640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3bf815bcb0 con 0x7f3bf8006ad0 2024-03-29T10:44:53.236 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:53.252+0000 7f3c071e8640 10 monclient(hunting): _renew_subs 2024-03-29T10:44:53.236 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:53.252+0000 7f3c071e8640 10 monclient(hunting): authenticate will time out at 1015.488165s 2024-03-29T10:44:53.241 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:53.257+0000 7f3c05785640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f3bf80cb780 0x7f3bf8162b50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:39535/0 (socket says 172.21.15.88:39535) 2024-03-29T10:44:53.241 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:53.257+0000 7f3c05785640 1 -- 172.21.15.88:0/1202093424 learned_addr learned my addr 172.21.15.88:0/1202093424 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-29T10:44:53.241 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:53.257+0000 7f3bfd7fa640 10 client.?.objecter ms_handle_connect 0x7f3bf80cb780 2024-03-29T10:44:53.242 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:53.257+0000 7f3bfd7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:44:53.242 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:53.257+0000 7f3bfd7fa640 1 -- 172.21.15.88:0/1202093424 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 228271950 0 0) 0x7f3bf8030220 con 0x7f3bf80cb780 2024-03-29T10:44:53.242 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:53.257+0000 7f3bfd7fa640 10 monclient(hunting): _negotiate 2024-03-29T10:44:53.242 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:53.257+0000 7f3bfd7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:53.242 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:53.258+0000 7f3bfd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:53.242 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:53.258+0000 7f3bfd7fa640 10 monclient(hunting): my global_id is 4368 2024-03-29T10:44:53.242 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:53.258+0000 7f3bfd7fa640 1 -- 172.21.15.88:0/1202093424 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3be4004eb0 con 0x7f3bf80cb780 2024-03-29T10:44:53.242 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:53.258+0000 7f3bfd7fa640 10 client.?.objecter ms_handle_connect 0x7f3bf80c5120 2024-03-29T10:44:53.242 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:53.258+0000 7f3bfd7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:44:53.242 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:53.258+0000 7f3bfd7fa640 1 -- 172.21.15.88:0/1202093424 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3072204387 0 0) 0x7f3bf80cab10 con 0x7f3bf80c5120 2024-03-29T10:44:53.242 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:53.258+0000 7f3bfd7fa640 10 monclient(hunting): _negotiate 2024-03-29T10:44:53.242 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:53.258+0000 7f3bfd7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:53.242 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:53.258+0000 7f3bfd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:53.243 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:53.258+0000 7f3bfd7fa640 10 monclient(hunting): my global_id is 4369 2024-03-29T10:44:53.243 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:53.258+0000 7f3bfd7fa640 1 -- 172.21.15.88:0/1202093424 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3bf8030220 con 0x7f3bf80c5120 2024-03-29T10:44:53.243 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:53.260+0000 7f3bfd7fa640 1 -- 172.21.15.88:0/1202093424 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 284373084 0 0) 0x7f3bf8030220 con 0x7f3bf80c5120 2024-03-29T10:44:53.243 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:53.260+0000 7f3bfd7fa640 1 -- 172.21.15.88:0/1202093424 --> v1:172.21.15.196:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3be4005490 con 0x7f3bf80c5120 2024-03-29T10:44:53.243 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:53.260+0000 7f3bfd7fa640 10 client.?.objecter ms_handle_connect 0x7f3bf8006ad0 2024-03-29T10:44:53.243 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:53.260+0000 7f3bfd7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:44:53.243 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:53.260+0000 7f3bfd7fa640 1 -- 172.21.15.88:0/1202093424 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f3bec001950 con 0x7f3bf80c5120 2024-03-29T10:44:53.244 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:53.260+0000 7f3bfd7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:53.244 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:53.260+0000 7f3bfd7fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:44:53.244 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:53.260+0000 7f3bfd7fa640 10 monclient(hunting): dump: 2024-03-29T10:44:53.244 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:44:53.244 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:44:53.244 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:44:53.244 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:44:53.244 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:53.244 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:44:53.244 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:44:53.244 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:44:53.244 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:44:53.244 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:44:53.244 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:53.260+0000 7f3bfd7fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:44:53.245 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:53.261+0000 7f3bfd7fa640 1 -- 172.21.15.88:0/1202093424 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1711230695 0 0) 0x7f3bf815bcb0 con 0x7f3bf8006ad0 2024-03-29T10:44:53.245 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:53.261+0000 7f3bfd7fa640 10 monclient(hunting): _negotiate 2024-03-29T10:44:53.245 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:53.261+0000 7f3bfd7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:53.245 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:53.261+0000 7f3bfd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:53.245 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:53.261+0000 7f3bfd7fa640 10 monclient(hunting): my global_id is 4397 2024-03-29T10:44:53.246 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:53.261+0000 7f3bfd7fa640 1 -- 172.21.15.88:0/1202093424 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3be4005950 con 0x7f3bf8006ad0 2024-03-29T10:44:53.246 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:53.261+0000 7f3bfd7fa640 1 -- 172.21.15.88:0/1202093424 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 306253919 0 0) 0x7f3be4004eb0 con 0x7f3bf80cb780 2024-03-29T10:44:53.246 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:53.261+0000 7f3bfd7fa640 1 -- 172.21.15.88:0/1202093424 --> v1:172.21.15.88:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3be4005de0 con 0x7f3bf80cb780 2024-03-29T10:44:53.246 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:53.261+0000 7f3bfd7fa640 1 -- 172.21.15.88:0/1202093424 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f3c00068810 con 0x7f3bf80cb780 2024-03-29T10:44:53.246 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:53.261+0000 7f3bfd7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:53.246 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:53.262+0000 7f3bfd7fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:44:53.246 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:53.262+0000 7f3bfd7fa640 10 monclient(hunting): dump: 2024-03-29T10:44:53.246 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:44:53.246 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:44:53.246 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:44:53.246 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:44:53.246 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:53.246 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:44:53.248 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:44:53.248 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:44:53.248 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:44:53.248 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:44:53.248 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:53.262+0000 7f3bfd7fa640 1 -- 172.21.15.88:0/1202093424 <== mon.1 v1:172.21.15.196:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 306993510 0 0) 0x7f3be4005490 con 0x7f3bf80c5120 2024-03-29T10:44:53.248 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:53.262+0000 7f3bfd7fa640 1 -- 172.21.15.88:0/1202093424 >> v1:172.21.15.88:6790/0 conn(0x7f3bf8006ad0 legacy=0x7f3bf815b5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:53.248 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:53.262+0000 7f3bfd7fa640 1 -- 172.21.15.88:0/1202093424 >> v1:172.21.15.88:6789/0 conn(0x7f3bf80cb780 legacy=0x7f3bf8162b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:53.248 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:53.262+0000 7f3bfd7fa640 10 monclient: _finish_hunting 0 2024-03-29T10:44:53.249 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:53.262+0000 7f3bfd7fa640 1 monclient: found mon.b 2024-03-29T10:44:53.249 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:53.262+0000 7f3bfd7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:44:53.249 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:53.262+0000 7f3bfd7fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T10:44:53.249 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:53.262+0000 7f3bfd7fa640 1 -- 172.21.15.88:0/1202093424 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bf80bd900 con 0x7f3bf80c5120 2024-03-29T10:44:53.249 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:53.262+0000 7f3bfd7fa640 10 monclient: _finish_auth 0 2024-03-29T10:44:53.249 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:53.262+0000 7f3bfd7fa640 10 monclient: _check_auth_tickets 2024-03-29T10:44:53.249 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:53.262+0000 7f3bfd7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:44:53.249 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:53.262+0000 7f3c071e8640 5 monclient: authenticate success, global_id 4369 2024-03-29T10:44:53.249 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:53.262+0000 7f3c071e8640 10 monclient: _renew_subs 2024-03-29T10:44:53.249 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:53.262+0000 7f3c071e8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T10:44:53.249 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:53.262+0000 7f3c071e8640 1 -- 172.21.15.88:0/1202093424 --> v1:172.21.15.196:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3bf80bdbb0 con 0x7f3bf80c5120 2024-03-29T10:44:53.249 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:53.262+0000 7f3c071e8640 10 client.4369.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:44:53.249 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:53.262+0000 7f3c071e8640 10 monclient: _renew_subs 2024-03-29T10:44:53.250 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:53.262+0000 7f3c071e8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T10:44:53.250 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:53.262+0000 7f3c071e8640 1 -- 172.21.15.88:0/1202093424 --> v1:172.21.15.196:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3bf8158990 con 0x7f3bf80c5120 2024-03-29T10:44:53.250 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:53.262+0000 7f3c071e8640 1 librados: init done 2024-03-29T10:44:53.252 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:53.263+0000 7f3c071e8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:44:53.252 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:53.264+0000 7f3c071e8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:44:53.252 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:53.264+0000 7f3c071e8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T10:44:53.252 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:53.264+0000 7f3c071e8640 1 -- 172.21.15.88:0/1202093424 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3bf815be40 con 0x7f3bf80c5120 2024-03-29T10:44:53.252 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:53.264+0000 7f3bfd7fa640 1 -- 172.21.15.88:0/1202093424 <== mon.1 v1:172.21.15.196:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3bec0047e0 con 0x7f3bf80c5120 2024-03-29T10:44:53.252 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:53.264+0000 7f3bfd7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:44:53.252 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:53.264+0000 7f3bfd7fa640 1 -- 172.21.15.88:0/1202093424 <== mon.1 v1:172.21.15.196:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f3bec001950 con 0x7f3bf80c5120 2024-03-29T10:44:53.253 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:53.264+0000 7f3bfd7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:53.253 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:53.264+0000 7f3bfd7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:44:53.253 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:53.264+0000 7f3bfd7fa640 10 monclient: dump: 2024-03-29T10:44:53.253 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:44:53.253 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:44:53.253 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:44:53.253 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:44:53.253 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:53.253 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:44:53.253 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:44:53.253 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:44:53.253 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:44:53.253 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:44:53.253 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:53.265+0000 7f3bfd7fa640 1 -- 172.21.15.88:0/1202093424 <== mon.1 v1:172.21.15.196:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4235629220 0 0) 0x7f3bec004ad0 con 0x7f3bf80c5120 2024-03-29T10:44:53.254 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:53.266+0000 7f3bfeffd640 4 set_mon_vals no callback set 2024-03-29T10:44:53.254 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:53.266+0000 7f3bfd7fa640 1 -- 172.21.15.88:0/1202093424 <== mon.1 v1:172.21.15.196:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4257673582 0 0) 0x7f3bec0544e0 con 0x7f3bf80c5120 2024-03-29T10:44:53.254 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:53.266+0000 7f3bfd7fa640 10 client.4369.objecter ms_dispatch 0x7f3c00000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:44:53.254 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:53.266+0000 7f3bfd7fa640 3 client.4369.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:44:53.254 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:53.266+0000 7f3bfd7fa640 3 client.4369.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:44:53.254 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:53.266+0000 7f3bfd7fa640 20 client.4369.objecter dump_active .. 0 homeless 2024-03-29T10:44:53.254 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:53.267+0000 7f3c04f84640 10 monclient: get_auth_request con 0x7f3be406e530 auth_method 0 2024-03-29T10:44:53.254 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:53.268+0000 7f3bfd7fa640 10 client.4369.objecter ms_handle_connect 0x7f3be406e530 2024-03-29T10:44:53.259 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:53.272+0000 7f3bfd7fa640 1 -- 172.21.15.88:0/1202093424 <== mon.1 v1:172.21.15.196:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f3bec04fe20 con 0x7f3bf80c5120 2024-03-29T10:44:53.259 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:53.272+0000 7f3bfd7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:44:53.259 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:53.272+0000 7f3bfd7fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:44:53.574 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:53.591+0000 7f3c071e8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:44:53.574 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:53.591+0000 7f3c071e8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:44:53.574 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:53.591+0000 7f3c071e8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T10:44:53.574 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:53.591+0000 7f3c071e8640 1 -- 172.21.15.88:0/1202093424 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3bf8004cd0 con 0x7f3bf80c5120 2024-03-29T10:44:53.575 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:53.592+0000 7f3bfd7fa640 1 -- 172.21.15.88:0/1202093424 <== mon.1 v1:172.21.15.196:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 344506374) 0x7f3bec051f20 con 0x7f3bf80c5120 2024-03-29T10:44:53.575 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:53.592+0000 7f3bfd7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:44:53.575 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:53.592+0000 7f3bfd7fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:44:53.577 INFO:teuthology.misc.health.smithi088.stdout: 2024-03-29T10:44:53.577 INFO:teuthology.misc.health.smithi088.stdout:{"epoch":10,"fsid":"869f5188-f92b-42dd-a46d-8c646e0388a9","created":"2024-03-29T10:40:40.285087+0000","modified":"2024-03-29T10:40:42.536230+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:40:42.536230+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a26c0c1c-8de0-438c-a3f6-3df10ea61574","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5de24915-8dec-49b6-8b16-f3170197c591","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ac054af3-d589-4565-9644-e6af75975c53","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3ec4f045-c714-4a22-bd1a-e07330b45ab3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"cddb9867-8e30-4951-9b13-3d1e78b9c45a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dc6d9911-785d-432f-8b53-56a1ccab302f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"77c88437-b587-496e-8b4d-1824ea8b5305","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ce963d96-5a8e-44c3-99de-c0dbebe234ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:44:53.579 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:53.596+0000 7f3c071e8640 10 librados: watch_flush enter 2024-03-29T10:44:53.579 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:53.596+0000 7f3c071e8640 10 librados: watch_flush exit 2024-03-29T10:44:53.579 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:53.596+0000 7f3c071e8640 20 client.4369.objecter shutdown clearing up homeless session... 2024-03-29T10:44:53.579 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:53.596+0000 7f3c071e8640 10 client.4369.objecter successfully canceled tick 2024-03-29T10:44:53.580 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:53.597+0000 7f3c071e8640 1 -- 172.21.15.88:0/1202093424 >> v1:172.21.15.88:6816/1752861349 conn(0x7f3be406e530 legacy=0x7f3be40709f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:53.580 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:53.597+0000 7f3c071e8640 10 monclient: shutdown 2024-03-29T10:44:53.580 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:53.597+0000 7f3c071e8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:44:53.580 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:53.597+0000 7f3c071e8640 1 -- 172.21.15.88:0/1202093424 >> v1:172.21.15.196:6789/0 conn(0x7f3bf80c5120 legacy=0x7f3bf81580a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:53.581 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:53.599+0000 7f3c071e8640 1 -- 172.21.15.88:0/1202093424 shutdown_connections 2024-03-29T10:44:53.582 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:53.599+0000 7f3c071e8640 1 -- 172.21.15.88:0/1202093424 >> 172.21.15.88:0/1202093424 conn(0x7f3bf80058f0 msgr2=0x7f3bf80043d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:44:53.582 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:53.599+0000 7f3c071e8640 1 -- 172.21.15.88:0/1202093424 shutdown_connections 2024-03-29T10:44:53.582 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:53.599+0000 7f3c071e8640 1 -- 172.21.15.88:0/1202093424 wait complete. 2024-03-29T10:44:53.583 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:44:53.599+0000 7f3c071e8640 1 librados: shutdown 2024-03-29T10:44:53.600 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:45:00.406 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:45:00.550 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:00.567+0000 7f617b5e2640 10 monclient: get_monmap_and_config 2024-03-29T10:45:00.551 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:00.567+0000 7f617b5e2640 10 monclient: build_initial_monmap 2024-03-29T10:45:00.551 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:00.567+0000 7f617b5e2640 10 monclient: monmap: 2024-03-29T10:45:00.551 INFO:teuthology.misc.health.smithi088.stderr:epoch 0 2024-03-29T10:45:00.551 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:45:00.551 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:45:00.568881+0000 2024-03-29T10:45:00.551 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:45:00.568881+0000 2024-03-29T10:45:00.551 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 0 (unknown) 2024-03-29T10:45:00.551 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:45:00.551 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-03-29T10:45:00.551 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-29T10:45:00.551 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-b 2024-03-29T10:45:00.551 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:45:00.552 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:00.568+0000 7f617b5e2640 1 Processor -- start 2024-03-29T10:45:00.552 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:00.569+0000 7f617b5e2640 1 -- start start 2024-03-29T10:45:00.552 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:00.569+0000 7f617b5e2640 10 monclient: init 2024-03-29T10:45:00.555 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:00.571+0000 7f617b5e2640 10 monclient: _reopen_session rank -1 2024-03-29T10:45:00.555 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:00.571+0000 7f617b5e2640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:45:00.555 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:00.571+0000 7f617b5e2640 10 monclient(hunting): picked mon.noname-b con 0x7f617406ebd0 addr v1:172.21.15.88:6790/0 2024-03-29T10:45:00.555 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:00.571+0000 7f617b5e2640 10 monclient(hunting): picked mon.noname-c con 0x7f617406efb0 addr v1:172.21.15.196:6789/0 2024-03-29T10:45:00.555 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:00.571+0000 7f617b5e2640 10 monclient(hunting): picked mon.noname-a con 0x7f617413ed90 addr v1:172.21.15.88:6789/0 2024-03-29T10:45:00.555 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:00.571+0000 7f617b5e2640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6174070b40 con 0x7f617413ed90 2024-03-29T10:45:00.555 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:00.571+0000 7f6178b7d640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f617406efb0 0x7f617413e680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.88:48930/0 (socket says 172.21.15.88:48930) 2024-03-29T10:45:00.555 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:00.571+0000 7f6178b7d640 1 -- 172.21.15.88:0/3088370560 learned_addr learned my addr 172.21.15.88:0/3088370560 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-29T10:45:00.555 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:00.572+0000 7f617b5e2640 1 -- 172.21.15.88:0/3088370560 --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6174070e20 con 0x7f617406efb0 2024-03-29T10:45:00.555 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:00.572+0000 7f617b5e2640 1 -- 172.21.15.88:0/3088370560 --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6174071100 con 0x7f617406ebd0 2024-03-29T10:45:00.555 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:00.572+0000 7f617b5e2640 10 monclient(hunting): _renew_subs 2024-03-29T10:45:00.555 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:00.572+0000 7f617b5e2640 10 monclient(hunting): authenticate will time out at 1022.808216s 2024-03-29T10:45:00.560 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:00.572+0000 7f616bfff640 1 -- 172.21.15.88:0/3088370560 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4166812131 0 0) 0x7f6174070b40 con 0x7f617413ed90 2024-03-29T10:45:00.560 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:00.572+0000 7f616bfff640 10 monclient(hunting): _negotiate 2024-03-29T10:45:00.560 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:00.572+0000 7f616bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:45:00.560 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:00.572+0000 7f616bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:45:00.560 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:00.573+0000 7f616bfff640 10 monclient(hunting): my global_id is 4371 2024-03-29T10:45:00.560 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:00.573+0000 7f616bfff640 1 -- 172.21.15.88:0/3088370560 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6160003520 con 0x7f617413ed90 2024-03-29T10:45:00.560 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:00.573+0000 7f616bfff640 1 -- 172.21.15.88:0/3088370560 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1796190527 0 0) 0x7f6174070e20 con 0x7f617406efb0 2024-03-29T10:45:00.560 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:00.573+0000 7f616bfff640 10 monclient(hunting): _negotiate 2024-03-29T10:45:00.561 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:00.573+0000 7f616bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:45:00.561 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:00.573+0000 7f616bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:45:00.561 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:00.573+0000 7f616bfff640 10 monclient(hunting): my global_id is 4372 2024-03-29T10:45:00.561 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:00.574+0000 7f616bfff640 1 -- 172.21.15.88:0/3088370560 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6174070b40 con 0x7f617406efb0 2024-03-29T10:45:00.561 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:00.574+0000 7f616bfff640 1 -- 172.21.15.88:0/3088370560 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 441726438 0 0) 0x7f6174071100 con 0x7f617406ebd0 2024-03-29T10:45:00.561 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:00.574+0000 7f616bfff640 10 monclient(hunting): _negotiate 2024-03-29T10:45:00.561 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:00.574+0000 7f616bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:45:00.561 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:00.574+0000 7f616bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:45:00.561 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:00.574+0000 7f616bfff640 10 monclient(hunting): my global_id is 4400 2024-03-29T10:45:00.561 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:00.574+0000 7f616bfff640 1 -- 172.21.15.88:0/3088370560 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6174070e20 con 0x7f617406ebd0 2024-03-29T10:45:00.561 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:00.574+0000 7f616bfff640 1 -- 172.21.15.88:0/3088370560 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2459539232 0 0) 0x7f6160003520 con 0x7f617413ed90 2024-03-29T10:45:00.561 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:00.574+0000 7f616bfff640 1 -- 172.21.15.88:0/3088370560 >> v1:172.21.15.88:6790/0 conn(0x7f617406ebd0 legacy=0x7f617413b930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:00.561 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:00.574+0000 7f616bfff640 1 -- 172.21.15.88:0/3088370560 >> v1:172.21.15.196:6789/0 conn(0x7f617406efb0 legacy=0x7f617413e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:00.561 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:00.574+0000 7f616bfff640 10 monclient: _finish_hunting 0 2024-03-29T10:45:00.561 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:00.574+0000 7f616bfff640 1 monclient: found mon.noname-a 2024-03-29T10:45:00.562 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:00.575+0000 7f616bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:45:00.562 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:00.575+0000 7f616bfff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.88:6789/0 2024-03-29T10:45:00.562 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:00.575+0000 7f616bfff640 1 -- 172.21.15.88:0/3088370560 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f617413c040 con 0x7f617413ed90 2024-03-29T10:45:00.562 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:00.575+0000 7f616bfff640 10 monclient: _finish_auth 0 2024-03-29T10:45:00.562 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:00.575+0000 7f616bfff640 10 monclient: _check_auth_tickets 2024-03-29T10:45:00.562 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:00.575+0000 7f616bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:45:00.562 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:00.575+0000 7f616bfff640 1 -- 172.21.15.88:0/3088370560 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f616c001a10 con 0x7f617413ed90 2024-03-29T10:45:00.562 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:00.575+0000 7f616bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:00.562 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:00.575+0000 7f616bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:45:00.562 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:00.575+0000 7f616bfff640 10 monclient: dump: 2024-03-29T10:45:00.562 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:45:00.562 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:45:00.562 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:45:00.563 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:45:00.563 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:00.563 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:45:00.563 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:45:00.563 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:45:00.563 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:45:00.563 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:45:00.563 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:00.575+0000 7f617b5e2640 5 monclient: authenticate success, global_id 4371 2024-03-29T10:45:00.563 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:00.575+0000 7f617b5e2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:45:00.563 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:00.576+0000 7f616bfff640 1 -- 172.21.15.88:0/3088370560 <== mon.0 v1:172.21.15.88:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f616c001c30 con 0x7f617413ed90 2024-03-29T10:45:00.563 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:00.576+0000 7f616bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:45:00.563 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:00.576+0000 7f616bfff640 1 -- 172.21.15.88:0/3088370560 <== mon.0 v1:172.21.15.88:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f616c001fc0 con 0x7f617413ed90 2024-03-29T10:45:00.563 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:00.576+0000 7f616bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:00.563 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:00.576+0000 7f616bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:45:00.564 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:00.576+0000 7f616bfff640 10 monclient: dump: 2024-03-29T10:45:00.564 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:45:00.564 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:45:00.564 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:45:00.564 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:45:00.564 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:00.564 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:45:00.564 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:45:00.564 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:45:00.564 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:45:00.564 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:45:00.564 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:00.576+0000 7f617b5e2640 10 monclient: get_monmap_and_config success 2024-03-29T10:45:00.564 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:00.576+0000 7f617b5e2640 4 set_mon_vals no callback set 2024-03-29T10:45:00.564 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:00.577+0000 7f617b5e2640 10 monclient: shutdown 2024-03-29T10:45:00.565 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:00.577+0000 7f617b5e2640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:45:00.565 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:00.577+0000 7f617b5e2640 1 -- 172.21.15.88:0/3088370560 >> v1:172.21.15.88:6789/0 conn(0x7f617413ed90 legacy=0x7f6174149d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:00.565 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:00.577+0000 7f617b5e2640 1 -- 172.21.15.88:0/3088370560 shutdown_connections 2024-03-29T10:45:00.565 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:00.577+0000 7f617b5e2640 1 -- 172.21.15.88:0/3088370560 >> 172.21.15.88:0/3088370560 conn(0x7f61740b4200 msgr2=0x7f61740b4610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:45:00.573 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:00.578+0000 7f617b5e2640 1 -- 172.21.15.88:0/3088370560 shutdown_connections 2024-03-29T10:45:00.573 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:00.578+0000 7f617b5e2640 1 -- 172.21.15.88:0/3088370560 wait complete. 2024-03-29T10:45:00.573 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:00.579+0000 7f617b5e2640 10 monclient: build_initial_monmap 2024-03-29T10:45:00.573 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:00.579+0000 7f617b5e2640 10 monclient: monmap: 2024-03-29T10:45:00.573 INFO:teuthology.misc.health.smithi088.stderr:epoch 0 2024-03-29T10:45:00.573 INFO:teuthology.misc.health.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:45:00.573 INFO:teuthology.misc.health.smithi088.stderr:last_changed 0.000000 2024-03-29T10:45:00.573 INFO:teuthology.misc.health.smithi088.stderr:created 0.000000 2024-03-29T10:45:00.574 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 0 (unknown) 2024-03-29T10:45:00.574 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:45:00.574 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-03-29T10:45:00.574 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-29T10:45:00.574 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-c 2024-03-29T10:45:00.574 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:45:00.574 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:00.579+0000 7f617b5e2640 1 librados: starting msgr at 2024-03-29T10:45:00.574 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:00.579+0000 7f617b5e2640 1 librados: starting objecter 2024-03-29T10:45:00.574 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:00.580+0000 7f617b5e2640 1 Processor -- start 2024-03-29T10:45:00.574 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:00.580+0000 7f617b5e2640 1 -- start start 2024-03-29T10:45:00.575 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:00.580+0000 7f617b5e2640 1 librados: setting wanted keys 2024-03-29T10:45:00.575 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:00.580+0000 7f617b5e2640 1 librados: calling monclient init 2024-03-29T10:45:00.575 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:00.580+0000 7f617b5e2640 10 monclient: init 2024-03-29T10:45:00.575 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:00.581+0000 7f617b5e2640 10 monclient: _reopen_session rank -1 2024-03-29T10:45:00.575 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:00.582+0000 7f617b5e2640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:45:00.575 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:00.582+0000 7f617b5e2640 10 monclient(hunting): picked mon.noname-a con 0x7f617406ebd0 addr v1:172.21.15.88:6789/0 2024-03-29T10:45:00.575 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:00.582+0000 7f617b5e2640 10 monclient(hunting): picked mon.noname-c con 0x7f617406efb0 addr v1:172.21.15.88:6790/0 2024-03-29T10:45:00.575 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:00.582+0000 7f617937e640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f617406ebd0 0x7f6174130730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:41024/0 (socket says 172.21.15.88:41024) 2024-03-29T10:45:00.575 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:00.582+0000 7f617937e640 1 -- 172.21.15.88:0/3296912586 learned_addr learned my addr 172.21.15.88:0/3296912586 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-29T10:45:00.575 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:00.582+0000 7f617b5e2640 10 monclient(hunting): picked mon.noname-b con 0x7f617413ed90 addr v1:172.21.15.196:6789/0 2024-03-29T10:45:00.575 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:00.582+0000 7f6169ffb640 10 client.?.objecter ms_handle_connect 0x7f617406ebd0 2024-03-29T10:45:00.575 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:00.582+0000 7f6169ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:45:00.575 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:00.583+0000 7f6169ffb640 10 client.?.objecter ms_handle_connect 0x7f617406efb0 2024-03-29T10:45:00.576 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:00.583+0000 7f6169ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:45:00.576 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:00.583+0000 7f6169ffb640 10 client.?.objecter ms_handle_connect 0x7f617413ed90 2024-03-29T10:45:00.576 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:00.583+0000 7f617b5e2640 1 -- 172.21.15.88:0/3296912586 --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f617412f080 con 0x7f617406ebd0 2024-03-29T10:45:00.576 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:00.583+0000 7f6169ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:45:00.576 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:00.583+0000 7f617b5e2640 1 -- 172.21.15.88:0/3296912586 --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6174131ef0 con 0x7f617413ed90 2024-03-29T10:45:00.576 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:00.583+0000 7f617b5e2640 1 -- 172.21.15.88:0/3296912586 --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61741321d0 con 0x7f617406efb0 2024-03-29T10:45:00.576 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:00.583+0000 7f617b5e2640 10 monclient(hunting): _renew_subs 2024-03-29T10:45:00.576 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:00.583+0000 7f617b5e2640 10 monclient(hunting): authenticate will time out at 1022.819698s 2024-03-29T10:45:00.576 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:00.584+0000 7f6169ffb640 1 -- 172.21.15.88:0/3296912586 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2567967991 0 0) 0x7f617412f080 con 0x7f617406ebd0 2024-03-29T10:45:00.576 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:00.584+0000 7f6169ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:45:00.576 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:00.584+0000 7f6169ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:45:00.576 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:00.584+0000 7f6169ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:45:00.576 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:00.584+0000 7f6169ffb640 10 monclient(hunting): my global_id is 4374 2024-03-29T10:45:00.576 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:00.584+0000 7f6169ffb640 1 -- 172.21.15.88:0/3296912586 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6158004eb0 con 0x7f617406ebd0 2024-03-29T10:45:00.576 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:00.584+0000 7f6169ffb640 1 -- 172.21.15.88:0/3296912586 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3704449507 0 0) 0x7f6174131ef0 con 0x7f617413ed90 2024-03-29T10:45:00.577 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:00.585+0000 7f6169ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:45:00.577 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:00.585+0000 7f6169ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:45:00.577 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:00.585+0000 7f6169ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:45:00.577 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:00.585+0000 7f6169ffb640 10 monclient(hunting): my global_id is 4375 2024-03-29T10:45:00.577 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:00.585+0000 7f6169ffb640 1 -- 172.21.15.88:0/3296912586 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f617412f080 con 0x7f617413ed90 2024-03-29T10:45:00.577 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:00.585+0000 7f6169ffb640 1 -- 172.21.15.88:0/3296912586 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1225862762 0 0) 0x7f61741321d0 con 0x7f617406efb0 2024-03-29T10:45:00.577 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:00.585+0000 7f6169ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:45:00.577 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:00.585+0000 7f6169ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:45:00.577 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:00.585+0000 7f6169ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:45:00.577 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:00.585+0000 7f6169ffb640 10 monclient(hunting): my global_id is 4403 2024-03-29T10:45:00.577 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:00.585+0000 7f6169ffb640 1 -- 172.21.15.88:0/3296912586 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6174131ef0 con 0x7f617406efb0 2024-03-29T10:45:00.577 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:00.585+0000 7f6169ffb640 1 -- 172.21.15.88:0/3296912586 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 7114726 0 0) 0x7f6158004eb0 con 0x7f617406ebd0 2024-03-29T10:45:00.577 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:00.585+0000 7f6169ffb640 1 -- 172.21.15.88:0/3296912586 --> v1:172.21.15.88:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f61741321d0 con 0x7f617406ebd0 2024-03-29T10:45:00.577 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:00.585+0000 7f6169ffb640 1 -- 172.21.15.88:0/3296912586 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f6170001930 con 0x7f617406ebd0 2024-03-29T10:45:00.578 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:00.585+0000 7f6169ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:00.578 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:00.585+0000 7f6169ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:45:00.578 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:00.585+0000 7f6169ffb640 10 monclient(hunting): dump: 2024-03-29T10:45:00.578 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:45:00.578 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:45:00.578 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:45:00.578 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:45:00.578 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:00.578 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:45:00.578 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:45:00.578 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:45:00.578 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:45:00.578 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:45:00.578 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:00.585+0000 7f6169ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:45:00.579 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:00.585+0000 7f6169ffb640 1 -- 172.21.15.88:0/3296912586 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2790950913 0 0) 0x7f617412f080 con 0x7f617413ed90 2024-03-29T10:45:00.579 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:00.585+0000 7f6169ffb640 1 -- 172.21.15.88:0/3296912586 --> v1:172.21.15.196:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6158004eb0 con 0x7f617413ed90 2024-03-29T10:45:00.579 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:00.585+0000 7f6169ffb640 1 -- 172.21.15.88:0/3296912586 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f616c001f40 con 0x7f617413ed90 2024-03-29T10:45:00.579 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:00.585+0000 7f6169ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:00.579 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:00.585+0000 7f6169ffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T10:45:00.579 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:00.585+0000 7f6169ffb640 10 monclient(hunting): dump: 2024-03-29T10:45:00.579 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:45:00.579 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:45:00.579 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:45:00.579 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:45:00.579 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:00.579 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:45:00.579 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:45:00.579 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:45:00.579 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:45:00.580 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:45:00.580 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:00.586+0000 7f6169ffb640 1 -- 172.21.15.88:0/3296912586 <== mon.1 v1:172.21.15.196:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 734692389 0 0) 0x7f6158004eb0 con 0x7f617413ed90 2024-03-29T10:45:00.580 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:00.586+0000 7f6169ffb640 1 -- 172.21.15.88:0/3296912586 >> v1:172.21.15.88:6790/0 conn(0x7f617406efb0 legacy=0x7f6174138730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:00.580 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:00.586+0000 7f6169ffb640 1 -- 172.21.15.88:0/3296912586 >> v1:172.21.15.88:6789/0 conn(0x7f617406ebd0 legacy=0x7f6174130730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:00.580 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:00.586+0000 7f6169ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:45:00.580 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:00.586+0000 7f6169ffb640 1 monclient: found mon.b 2024-03-29T10:45:00.580 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:00.586+0000 7f6169ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:45:00.580 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:00.586+0000 7f6169ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T10:45:00.580 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:00.586+0000 7f6169ffb640 1 -- 172.21.15.88:0/3296912586 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6174130210 con 0x7f617413ed90 2024-03-29T10:45:00.580 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:00.586+0000 7f6169ffb640 10 monclient: _finish_auth 0 2024-03-29T10:45:00.580 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:00.586+0000 7f6169ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:45:00.580 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:00.586+0000 7f6169ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:45:00.580 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:00.587+0000 7f617b5e2640 5 monclient: authenticate success, global_id 4375 2024-03-29T10:45:00.580 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:00.587+0000 7f617b5e2640 10 monclient: _renew_subs 2024-03-29T10:45:00.581 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:00.587+0000 7f617b5e2640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T10:45:00.581 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:00.587+0000 7f617b5e2640 1 -- 172.21.15.88:0/3296912586 --> v1:172.21.15.196:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6174130e40 con 0x7f617413ed90 2024-03-29T10:45:00.581 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:00.587+0000 7f617b5e2640 10 client.4375.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:45:00.581 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:00.587+0000 7f617b5e2640 10 monclient: _renew_subs 2024-03-29T10:45:00.581 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:00.587+0000 7f617b5e2640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T10:45:00.581 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:00.587+0000 7f617b5e2640 1 -- 172.21.15.88:0/3296912586 --> v1:172.21.15.196:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f61741dac00 con 0x7f617413ed90 2024-03-29T10:45:00.581 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:00.587+0000 7f617b5e2640 1 librados: init done 2024-03-29T10:45:00.581 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:00.587+0000 7f6169ffb640 1 -- 172.21.15.88:0/3296912586 <== mon.1 v1:172.21.15.196:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f616c0020a0 con 0x7f617413ed90 2024-03-29T10:45:00.581 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:00.587+0000 7f6169ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:45:00.581 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:00.587+0000 7f616b7fe640 4 set_mon_vals no callback set 2024-03-29T10:45:00.581 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:00.588+0000 7f6169ffb640 1 -- 172.21.15.88:0/3296912586 <== mon.1 v1:172.21.15.196:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f616c003dc0 con 0x7f617413ed90 2024-03-29T10:45:00.581 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:00.588+0000 7f6169ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:00.581 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:00.588+0000 7f6169ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:45:00.581 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:00.588+0000 7f6169ffb640 10 monclient: dump: 2024-03-29T10:45:00.582 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:45:00.582 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:45:00.582 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:45:00.582 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:45:00.582 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:00.582 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:45:00.582 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:45:00.582 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:45:00.582 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:45:00.582 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:45:00.582 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:00.589+0000 7f617b5e2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:45:00.582 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:00.589+0000 7f617b5e2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:45:00.582 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:00.589+0000 7f617b5e2640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T10:45:00.582 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:00.589+0000 7f617b5e2640 1 -- 172.21.15.88:0/3296912586 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6154005080 con 0x7f617413ed90 2024-03-29T10:45:00.583 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:00.590+0000 7f6169ffb640 1 -- 172.21.15.88:0/3296912586 <== mon.1 v1:172.21.15.196:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4235629220 0 0) 0x7f616c019d90 con 0x7f617413ed90 2024-03-29T10:45:00.583 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:00.591+0000 7f617937e640 10 monclient: get_auth_request con 0x7f615806e000 auth_method 0 2024-03-29T10:45:00.583 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:00.591+0000 7f6169ffb640 1 -- 172.21.15.88:0/3296912586 <== mon.1 v1:172.21.15.196:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4257673582 0 0) 0x7f616c083b00 con 0x7f617413ed90 2024-03-29T10:45:00.583 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:00.591+0000 7f6169ffb640 10 client.4375.objecter ms_dispatch 0x7f6174000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:45:00.584 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:00.591+0000 7f6169ffb640 3 client.4375.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:45:00.584 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:00.591+0000 7f6169ffb640 3 client.4375.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:45:00.584 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:00.591+0000 7f6169ffb640 20 client.4375.objecter dump_active .. 0 homeless 2024-03-29T10:45:00.584 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:00.591+0000 7f6169ffb640 10 client.4375.objecter ms_handle_connect 0x7f615806e000 2024-03-29T10:45:00.584 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:00.597+0000 7f6169ffb640 1 -- 172.21.15.88:0/3296912586 <== mon.1 v1:172.21.15.196:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f616c04f3e0 con 0x7f617413ed90 2024-03-29T10:45:00.584 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:00.598+0000 7f6169ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:45:00.584 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:00.598+0000 7f6169ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:45:00.969 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:00.986+0000 7f617b5e2640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:45:00.969 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:00.986+0000 7f617b5e2640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:45:00.969 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:00.986+0000 7f617b5e2640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T10:45:00.969 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:00.986+0000 7f617b5e2640 1 -- 172.21.15.88:0/3296912586 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6154005330 con 0x7f617413ed90 2024-03-29T10:45:00.970 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:00.987+0000 7f6169ffb640 1 -- 172.21.15.88:0/3296912586 <== mon.1 v1:172.21.15.196:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 344506374) 0x7f616c051310 con 0x7f617413ed90 2024-03-29T10:45:00.970 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:00.987+0000 7f6169ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:45:00.970 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:00.987+0000 7f6169ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:45:00.970 INFO:teuthology.misc.health.smithi088.stdout: 2024-03-29T10:45:00.970 INFO:teuthology.misc.health.smithi088.stdout:{"epoch":10,"fsid":"869f5188-f92b-42dd-a46d-8c646e0388a9","created":"2024-03-29T10:40:40.285087+0000","modified":"2024-03-29T10:40:42.536230+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:40:42.536230+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a26c0c1c-8de0-438c-a3f6-3df10ea61574","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5de24915-8dec-49b6-8b16-f3170197c591","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ac054af3-d589-4565-9644-e6af75975c53","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3ec4f045-c714-4a22-bd1a-e07330b45ab3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"cddb9867-8e30-4951-9b13-3d1e78b9c45a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dc6d9911-785d-432f-8b53-56a1ccab302f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"77c88437-b587-496e-8b4d-1824ea8b5305","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ce963d96-5a8e-44c3-99de-c0dbebe234ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:45:00.974 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:00.991+0000 7f617b5e2640 10 librados: watch_flush enter 2024-03-29T10:45:00.974 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:00.991+0000 7f617b5e2640 10 librados: watch_flush exit 2024-03-29T10:45:00.974 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:00.991+0000 7f617b5e2640 20 client.4375.objecter shutdown clearing up homeless session... 2024-03-29T10:45:00.974 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:00.991+0000 7f617b5e2640 10 client.4375.objecter successfully canceled tick 2024-03-29T10:45:00.974 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:00.991+0000 7f617b5e2640 1 -- 172.21.15.88:0/3296912586 >> v1:172.21.15.88:6816/1752861349 conn(0x7f615806e000 legacy=0x7f61580704c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:00.974 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:00.991+0000 7f617b5e2640 10 monclient: shutdown 2024-03-29T10:45:00.974 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:00.991+0000 7f617b5e2640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:45:00.975 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:00.991+0000 7f617b5e2640 1 -- 172.21.15.88:0/3296912586 >> v1:172.21.15.196:6789/0 conn(0x7f617413ed90 legacy=0x7f61741e2d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:00.975 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:00.991+0000 7f617b5e2640 1 -- 172.21.15.88:0/3296912586 shutdown_connections 2024-03-29T10:45:00.975 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:00.991+0000 7f617b5e2640 1 -- 172.21.15.88:0/3296912586 >> 172.21.15.88:0/3296912586 conn(0x7f61740b4200 msgr2=0x7f61740743e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:45:00.975 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:00.992+0000 7f617b5e2640 1 -- 172.21.15.88:0/3296912586 shutdown_connections 2024-03-29T10:45:00.975 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:00.992+0000 7f617b5e2640 1 -- 172.21.15.88:0/3296912586 wait complete. 2024-03-29T10:45:00.975 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:00.992+0000 7f617b5e2640 1 librados: shutdown 2024-03-29T10:45:00.989 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:45:07.796 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:45:07.948 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:07.965+0000 7f622b59e640 10 monclient: get_monmap_and_config 2024-03-29T10:45:07.949 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:07.965+0000 7f622b59e640 10 monclient: build_initial_monmap 2024-03-29T10:45:07.949 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:07.965+0000 7f622b59e640 10 monclient: monmap: 2024-03-29T10:45:07.949 INFO:teuthology.misc.health.smithi088.stderr:epoch 0 2024-03-29T10:45:07.949 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:45:07.949 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:45:07.966814+0000 2024-03-29T10:45:07.949 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:45:07.966814+0000 2024-03-29T10:45:07.949 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 0 (unknown) 2024-03-29T10:45:07.949 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:45:07.949 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-03-29T10:45:07.949 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-29T10:45:07.949 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-b 2024-03-29T10:45:07.949 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:45:07.955 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:07.966+0000 7f622b59e640 1 Processor -- start 2024-03-29T10:45:07.955 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:07.966+0000 7f622b59e640 1 -- start start 2024-03-29T10:45:07.955 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:07.966+0000 7f622b59e640 10 monclient: init 2024-03-29T10:45:07.955 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:07.967+0000 7f622b59e640 10 monclient: _reopen_session rank -1 2024-03-29T10:45:07.955 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:07.967+0000 7f622b59e640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:45:07.955 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:07.968+0000 7f622b59e640 10 monclient(hunting): picked mon.noname-b con 0x7f622c0b4920 addr v1:172.21.15.88:6790/0 2024-03-29T10:45:07.955 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:07.968+0000 7f622b59e640 10 monclient(hunting): picked mon.noname-a con 0x7f622c0b4200 addr v1:172.21.15.88:6789/0 2024-03-29T10:45:07.955 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:07.968+0000 7f622b59e640 10 monclient(hunting): picked mon.noname-c con 0x7f622c13f9a0 addr v1:172.21.15.196:6789/0 2024-03-29T10:45:07.955 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:07.968+0000 7f622b59e640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f622c0b0930 con 0x7f622c0b4200 2024-03-29T10:45:07.955 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:07.968+0000 7f622b59e640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f622c0af460 con 0x7f622c13f9a0 2024-03-29T10:45:07.956 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:07.968+0000 7f622a59c640 1 --1- >> v1:172.21.15.88:6790/0 conn(0x7f622c0b4920 0x7f622c13f290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6790/0 says I am v1:172.21.15.88:39932/0 (socket says 172.21.15.88:39932) 2024-03-29T10:45:07.956 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:07.968+0000 7f622a59c640 1 -- 172.21.15.88:0/346266678 learned_addr learned my addr 172.21.15.88:0/346266678 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-29T10:45:07.956 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:07.968+0000 7f622b59e640 1 -- 172.21.15.88:0/346266678 --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f622c0aeda0 con 0x7f622c0b4920 2024-03-29T10:45:07.956 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:07.968+0000 7f622b59e640 10 monclient(hunting): _renew_subs 2024-03-29T10:45:07.956 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:07.968+0000 7f622b59e640 10 monclient(hunting): authenticate will time out at 1030.204211s 2024-03-29T10:45:07.956 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:07.969+0000 7f622959a640 1 -- 172.21.15.88:0/346266678 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3916251089 0 0) 0x7f622c0b0930 con 0x7f622c0b4200 2024-03-29T10:45:07.956 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:07.969+0000 7f622959a640 10 monclient(hunting): _negotiate 2024-03-29T10:45:07.956 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:07.969+0000 7f622959a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:45:07.956 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:07.969+0000 7f622959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:45:07.956 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:07.969+0000 7f622959a640 10 monclient(hunting): my global_id is 4377 2024-03-29T10:45:07.956 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:07.969+0000 7f622959a640 1 -- 172.21.15.88:0/346266678 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6214003520 con 0x7f622c0b4200 2024-03-29T10:45:07.956 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:07.969+0000 7f622959a640 1 -- 172.21.15.88:0/346266678 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1738452897 0 0) 0x7f622c0aeda0 con 0x7f622c0b4920 2024-03-29T10:45:07.956 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:07.969+0000 7f622959a640 10 monclient(hunting): _negotiate 2024-03-29T10:45:07.956 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:07.969+0000 7f622959a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:45:07.957 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:07.969+0000 7f622959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:45:07.957 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:07.969+0000 7f622959a640 10 monclient(hunting): my global_id is 4406 2024-03-29T10:45:07.957 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:07.969+0000 7f622959a640 1 -- 172.21.15.88:0/346266678 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f622c0b0930 con 0x7f622c0b4920 2024-03-29T10:45:07.957 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:07.969+0000 7f622959a640 1 -- 172.21.15.88:0/346266678 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 859048985 0 0) 0x7f622c0af460 con 0x7f622c13f9a0 2024-03-29T10:45:07.957 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:07.969+0000 7f622959a640 10 monclient(hunting): _negotiate 2024-03-29T10:45:07.957 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:07.969+0000 7f622959a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:45:07.957 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:07.969+0000 7f622959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:45:07.957 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:07.969+0000 7f622959a640 10 monclient(hunting): my global_id is 4378 2024-03-29T10:45:07.957 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:07.969+0000 7f622959a640 1 -- 172.21.15.88:0/346266678 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f622c0aeda0 con 0x7f622c13f9a0 2024-03-29T10:45:07.957 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:07.969+0000 7f622959a640 1 -- 172.21.15.88:0/346266678 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2729195638 0 0) 0x7f6214003520 con 0x7f622c0b4200 2024-03-29T10:45:07.957 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:07.969+0000 7f622959a640 1 -- 172.21.15.88:0/346266678 >> v1:172.21.15.88:6790/0 conn(0x7f622c0b4920 legacy=0x7f622c13f290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:07.957 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:07.970+0000 7f622959a640 1 -- 172.21.15.88:0/346266678 >> v1:172.21.15.196:6789/0 conn(0x7f622c13f9a0 legacy=0x7f622c14d8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:07.957 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:07.970+0000 7f622959a640 10 monclient: _finish_hunting 0 2024-03-29T10:45:07.957 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:07.970+0000 7f622959a640 1 monclient: found mon.noname-a 2024-03-29T10:45:07.958 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:07.970+0000 7f622959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:45:07.958 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:07.970+0000 7f622959a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.88:6789/0 2024-03-29T10:45:07.958 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:07.970+0000 7f622959a640 1 -- 172.21.15.88:0/346266678 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f622c0af980 con 0x7f622c0b4200 2024-03-29T10:45:07.958 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:07.970+0000 7f622959a640 10 monclient: _finish_auth 0 2024-03-29T10:45:07.958 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:07.970+0000 7f622959a640 10 monclient: _check_auth_tickets 2024-03-29T10:45:07.958 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:07.970+0000 7f622959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:45:07.958 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:07.970+0000 7f622959a640 1 -- 172.21.15.88:0/346266678 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f6220001690 con 0x7f622c0b4200 2024-03-29T10:45:07.958 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:07.970+0000 7f622959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:07.958 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:07.970+0000 7f622959a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:45:07.958 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:07.970+0000 7f622959a640 10 monclient: dump: 2024-03-29T10:45:07.958 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:45:07.959 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:45:07.959 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:45:07.959 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:45:07.959 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:07.959 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:45:07.959 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:45:07.959 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:45:07.959 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:45:07.959 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:45:07.959 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:07.970+0000 7f622b59e640 5 monclient: authenticate success, global_id 4377 2024-03-29T10:45:07.959 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:07.970+0000 7f622b59e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:45:07.959 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:07.970+0000 7f622959a640 1 -- 172.21.15.88:0/346266678 <== mon.0 v1:172.21.15.88:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f62200017f0 con 0x7f622c0b4200 2024-03-29T10:45:07.959 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:07.970+0000 7f622959a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:45:07.959 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:07.970+0000 7f622959a640 1 -- 172.21.15.88:0/346266678 <== mon.0 v1:172.21.15.88:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f6220001c00 con 0x7f622c0b4200 2024-03-29T10:45:07.960 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:07.970+0000 7f622b59e640 10 monclient: get_monmap_and_config success 2024-03-29T10:45:07.960 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:07.970+0000 7f622b59e640 4 set_mon_vals no callback set 2024-03-29T10:45:07.960 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:07.970+0000 7f622959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:07.960 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:07.971+0000 7f622959a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:45:07.960 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:07.971+0000 7f622959a640 10 monclient: dump: 2024-03-29T10:45:07.960 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:45:07.960 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:45:07.960 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:45:07.960 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:45:07.960 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:07.960 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:45:07.960 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:45:07.960 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:45:07.960 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:45:07.960 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:45:07.961 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:07.971+0000 7f622b59e640 10 monclient: shutdown 2024-03-29T10:45:07.961 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:07.971+0000 7f622b59e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:45:07.961 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:07.971+0000 7f622b59e640 1 -- 172.21.15.88:0/346266678 >> v1:172.21.15.88:6789/0 conn(0x7f622c0b4200 legacy=0x7f622c0b0fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:07.961 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:07.971+0000 7f622b59e640 1 -- 172.21.15.88:0/346266678 shutdown_connections 2024-03-29T10:45:07.961 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:07.971+0000 7f622b59e640 1 -- 172.21.15.88:0/346266678 >> 172.21.15.88:0/346266678 conn(0x7f622c06d230 msgr2=0x7f622c066a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:45:07.961 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:07.971+0000 7f622b59e640 1 -- 172.21.15.88:0/346266678 shutdown_connections 2024-03-29T10:45:07.961 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:07.972+0000 7f622b59e640 1 -- 172.21.15.88:0/346266678 wait complete. 2024-03-29T10:45:07.968 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:07.973+0000 7f622b59e640 10 monclient: build_initial_monmap 2024-03-29T10:45:07.968 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:07.973+0000 7f622b59e640 10 monclient: monmap: 2024-03-29T10:45:07.968 INFO:teuthology.misc.health.smithi088.stderr:epoch 0 2024-03-29T10:45:07.968 INFO:teuthology.misc.health.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:45:07.969 INFO:teuthology.misc.health.smithi088.stderr:last_changed 0.000000 2024-03-29T10:45:07.969 INFO:teuthology.misc.health.smithi088.stderr:created 0.000000 2024-03-29T10:45:07.969 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 0 (unknown) 2024-03-29T10:45:07.969 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:45:07.969 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-03-29T10:45:07.969 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-29T10:45:07.969 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-c 2024-03-29T10:45:07.969 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:45:07.969 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:07.974+0000 7f622b59e640 1 librados: starting msgr at 2024-03-29T10:45:07.969 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:07.974+0000 7f622b59e640 1 librados: starting objecter 2024-03-29T10:45:07.969 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:07.974+0000 7f622b59e640 1 Processor -- start 2024-03-29T10:45:07.969 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:07.974+0000 7f622b59e640 1 -- start start 2024-03-29T10:45:07.969 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:07.974+0000 7f622b59e640 1 librados: setting wanted keys 2024-03-29T10:45:07.969 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:07.974+0000 7f622b59e640 1 librados: calling monclient init 2024-03-29T10:45:07.970 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:07.974+0000 7f622b59e640 10 monclient: init 2024-03-29T10:45:07.970 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:07.975+0000 7f622b59e640 10 monclient: _reopen_session rank -1 2024-03-29T10:45:07.970 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:07.975+0000 7f622b59e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:45:07.970 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:07.975+0000 7f622b59e640 10 monclient(hunting): picked mon.noname-b con 0x7f622c0b4200 addr v1:172.21.15.196:6789/0 2024-03-29T10:45:07.970 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:07.975+0000 7f622b59e640 10 monclient(hunting): picked mon.noname-a con 0x7f622c0b4920 addr v1:172.21.15.88:6789/0 2024-03-29T10:45:07.970 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:07.975+0000 7f622b59e640 10 monclient(hunting): picked mon.noname-c con 0x7f622c13f9a0 addr v1:172.21.15.88:6790/0 2024-03-29T10:45:07.970 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:07.975+0000 7f622b59e640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f622c06f6b0 con 0x7f622c0b4920 2024-03-29T10:45:07.970 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:07.975+0000 7f622b59e640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f622c06f990 con 0x7f622c0b4200 2024-03-29T10:45:07.970 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:07.975+0000 7f6229d9b640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f622c0b4920 0x7f622c0abd30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:47256/0 (socket says 172.21.15.88:47256) 2024-03-29T10:45:07.970 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:07.975+0000 7f622ad9d640 1 --1- >> v1:172.21.15.88:6790/0 conn(0x7f622c13f9a0 0x7f622c0ac440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6790/0 says I am v1:172.21.15.88:40142/0 (socket says 172.21.15.88:40142) 2024-03-29T10:45:07.970 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:07.975+0000 7f6229d9b640 1 -- 172.21.15.88:0/2661893178 learned_addr learned my addr 172.21.15.88:0/2661893178 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-29T10:45:07.970 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:07.975+0000 7f622b59e640 1 -- 172.21.15.88:0/2661893178 --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f622c06fc70 con 0x7f622c13f9a0 2024-03-29T10:45:07.970 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:07.975+0000 7f622b59e640 10 monclient(hunting): _renew_subs 2024-03-29T10:45:07.971 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:07.975+0000 7f622b59e640 10 monclient(hunting): authenticate will time out at 1030.211699s 2024-03-29T10:45:07.971 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:07.975+0000 7f621affd640 10 client.?.objecter ms_handle_connect 0x7f622c0b4920 2024-03-29T10:45:07.971 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:07.975+0000 7f621affd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:45:07.971 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:07.975+0000 7f621affd640 10 client.?.objecter ms_handle_connect 0x7f622c13f9a0 2024-03-29T10:45:07.971 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:07.975+0000 7f621affd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:45:07.971 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:07.976+0000 7f621affd640 1 -- 172.21.15.88:0/2661893178 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4117478248 0 0) 0x7f622c06fc70 con 0x7f622c13f9a0 2024-03-29T10:45:07.971 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:07.976+0000 7f621affd640 10 monclient(hunting): _negotiate 2024-03-29T10:45:07.971 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:07.976+0000 7f621affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:45:07.971 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:07.976+0000 7f621affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:45:07.971 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:07.976+0000 7f621affd640 10 monclient(hunting): my global_id is 4409 2024-03-29T10:45:07.971 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:07.976+0000 7f621affd640 1 -- 172.21.15.88:0/2661893178 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f620c004eb0 con 0x7f622c13f9a0 2024-03-29T10:45:07.971 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:07.976+0000 7f621affd640 1 -- 172.21.15.88:0/2661893178 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3464164633 0 0) 0x7f622c06f6b0 con 0x7f622c0b4920 2024-03-29T10:45:07.971 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:07.976+0000 7f621affd640 10 monclient(hunting): _negotiate 2024-03-29T10:45:07.971 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:07.976+0000 7f621affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:45:07.972 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:07.977+0000 7f621affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:45:07.972 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:07.977+0000 7f621affd640 10 monclient(hunting): my global_id is 4380 2024-03-29T10:45:07.972 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:07.977+0000 7f621affd640 1 -- 172.21.15.88:0/2661893178 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f622c06fc70 con 0x7f622c0b4920 2024-03-29T10:45:07.972 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:07.977+0000 7f621affd640 10 client.?.objecter ms_handle_connect 0x7f622c0b4200 2024-03-29T10:45:07.972 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:07.977+0000 7f621affd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:45:07.972 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:07.977+0000 7f621affd640 1 -- 172.21.15.88:0/2661893178 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3643201080 0 0) 0x7f620c004eb0 con 0x7f622c13f9a0 2024-03-29T10:45:07.972 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:07.977+0000 7f621affd640 1 -- 172.21.15.88:0/2661893178 --> v1:172.21.15.88:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f622c06f6b0 con 0x7f622c13f9a0 2024-03-29T10:45:07.972 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:07.977+0000 7f621affd640 1 -- 172.21.15.88:0/2661893178 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f6224001bd0 con 0x7f622c13f9a0 2024-03-29T10:45:07.972 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:07.977+0000 7f621affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:07.972 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:07.977+0000 7f621affd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:45:07.972 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:07.977+0000 7f621affd640 10 monclient(hunting): dump: 2024-03-29T10:45:07.972 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:45:07.972 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:45:07.972 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:45:07.973 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:45:07.973 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:07.973 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:45:07.973 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:45:07.973 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:45:07.973 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:45:07.973 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:45:07.973 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:07.977+0000 7f621affd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:45:07.973 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:07.977+0000 7f621affd640 1 -- 172.21.15.88:0/2661893178 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 999306714 0 0) 0x7f622c06f990 con 0x7f622c0b4200 2024-03-29T10:45:07.973 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:07.977+0000 7f621affd640 10 monclient(hunting): _negotiate 2024-03-29T10:45:07.973 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:07.977+0000 7f621affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:45:07.973 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:07.977+0000 7f621affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:45:07.973 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:07.978+0000 7f621affd640 10 monclient(hunting): my global_id is 4381 2024-03-29T10:45:07.973 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:07.978+0000 7f621affd640 1 -- 172.21.15.88:0/2661893178 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f620c004eb0 con 0x7f622c0b4200 2024-03-29T10:45:07.973 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:07.978+0000 7f621affd640 1 -- 172.21.15.88:0/2661893178 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 488788151 0 0) 0x7f622c06fc70 con 0x7f622c0b4920 2024-03-29T10:45:07.974 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:07.978+0000 7f621affd640 1 -- 172.21.15.88:0/2661893178 --> v1:172.21.15.88:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f622c06f990 con 0x7f622c0b4920 2024-03-29T10:45:07.974 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:07.978+0000 7f621affd640 1 -- 172.21.15.88:0/2661893178 <== mon.2 v1:172.21.15.88:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3849674495 0 0) 0x7f622c06f6b0 con 0x7f622c13f9a0 2024-03-29T10:45:07.974 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:07.978+0000 7f621affd640 1 -- 172.21.15.88:0/2661893178 >> v1:172.21.15.196:6789/0 conn(0x7f622c0b4200 legacy=0x7f622c1db920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:07.974 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:07.978+0000 7f621affd640 1 -- 172.21.15.88:0/2661893178 >> v1:172.21.15.88:6789/0 conn(0x7f622c0b4920 legacy=0x7f622c0abd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:07.974 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:07.978+0000 7f621affd640 10 monclient: _finish_hunting 0 2024-03-29T10:45:07.974 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:07.978+0000 7f621affd640 1 monclient: found mon.c 2024-03-29T10:45:07.974 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:07.978+0000 7f621affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:45:07.974 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:07.978+0000 7f621affd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-03-29T10:45:07.974 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:07.978+0000 7f621affd640 1 -- 172.21.15.88:0/2661893178 --> v1:172.21.15.88:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f622c0ab810 con 0x7f622c13f9a0 2024-03-29T10:45:07.974 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:07.978+0000 7f621affd640 10 monclient: _finish_auth 0 2024-03-29T10:45:07.974 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:07.978+0000 7f621affd640 10 monclient: _check_auth_tickets 2024-03-29T10:45:07.974 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:07.978+0000 7f621affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:45:07.974 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:07.978+0000 7f622b59e640 5 monclient: authenticate success, global_id 4409 2024-03-29T10:45:07.974 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:07.978+0000 7f622b59e640 10 monclient: _renew_subs 2024-03-29T10:45:07.975 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:07.978+0000 7f622b59e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-03-29T10:45:07.975 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:07.978+0000 7f622b59e640 1 -- 172.21.15.88:0/2661893178 --> v1:172.21.15.88:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f622c06de90 con 0x7f622c13f9a0 2024-03-29T10:45:07.975 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:07.978+0000 7f622b59e640 10 client.4409.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:45:07.975 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:07.978+0000 7f622b59e640 10 monclient: _renew_subs 2024-03-29T10:45:07.975 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:07.978+0000 7f622b59e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-03-29T10:45:07.975 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:07.978+0000 7f622b59e640 1 -- 172.21.15.88:0/2661893178 --> v1:172.21.15.88:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f622c1dc210 con 0x7f622c13f9a0 2024-03-29T10:45:07.975 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:07.978+0000 7f622b59e640 1 librados: init done 2024-03-29T10:45:07.975 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:07.979+0000 7f621affd640 1 -- 172.21.15.88:0/2661893178 <== mon.2 v1:172.21.15.88:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6224003990 con 0x7f622c13f9a0 2024-03-29T10:45:07.975 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:07.979+0000 7f621affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:45:07.975 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:07.979+0000 7f6228d99640 4 set_mon_vals no callback set 2024-03-29T10:45:07.975 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:07.979+0000 7f621affd640 1 -- 172.21.15.88:0/2661893178 <== mon.2 v1:172.21.15.88:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f6224003ce0 con 0x7f622c13f9a0 2024-03-29T10:45:07.975 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:07.979+0000 7f621affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:07.975 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:07.979+0000 7f621affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:45:07.975 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:07.979+0000 7f621affd640 10 monclient: dump: 2024-03-29T10:45:07.976 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:45:07.976 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:45:07.976 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:45:07.976 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:45:07.976 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:07.976 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:45:07.976 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:45:07.976 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:45:07.976 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:45:07.976 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:45:07.976 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:07.980+0000 7f622b59e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:45:07.976 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:07.981+0000 7f621affd640 1 -- 172.21.15.88:0/2661893178 <== mon.2 v1:172.21.15.88:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4235629220 0 0) 0x7f6224003e40 con 0x7f622c13f9a0 2024-03-29T10:45:07.976 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:07.981+0000 7f622b59e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:45:07.976 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:07.981+0000 7f622b59e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-03-29T10:45:07.976 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:07.981+0000 7f622b59e640 1 -- 172.21.15.88:0/2661893178 --> v1:172.21.15.88:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6210005080 con 0x7f622c13f9a0 2024-03-29T10:45:07.977 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:07.982+0000 7f622a59c640 10 monclient: get_auth_request con 0x7f620c06ded0 auth_method 0 2024-03-29T10:45:07.977 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:07.983+0000 7f621affd640 1 -- 172.21.15.88:0/2661893178 <== mon.2 v1:172.21.15.88:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4257673582 0 0) 0x7f6224084ca0 con 0x7f622c13f9a0 2024-03-29T10:45:07.977 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:07.983+0000 7f621affd640 10 client.4409.objecter ms_dispatch 0x7f622c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:45:07.977 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:07.983+0000 7f621affd640 3 client.4409.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:45:07.977 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:07.983+0000 7f621affd640 3 client.4409.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:45:07.977 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:07.983+0000 7f621affd640 20 client.4409.objecter dump_active .. 0 homeless 2024-03-29T10:45:07.977 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:07.983+0000 7f621affd640 10 client.4409.objecter ms_handle_connect 0x7f620c06ded0 2024-03-29T10:45:07.977 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:07.992+0000 7f621affd640 1 -- 172.21.15.88:0/2661893178 <== mon.2 v1:172.21.15.88:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6224050580 con 0x7f622c13f9a0 2024-03-29T10:45:07.977 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:07.992+0000 7f621affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:45:07.978 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:07.992+0000 7f621affd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:45:08.368 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:08.385+0000 7f622b59e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:45:08.368 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:08.385+0000 7f622b59e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:45:08.368 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:08.385+0000 7f622b59e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-03-29T10:45:08.368 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:08.385+0000 7f622b59e640 1 -- 172.21.15.88:0/2661893178 --> v1:172.21.15.88:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6210005330 con 0x7f622c13f9a0 2024-03-29T10:45:08.369 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:08.386+0000 7f621affd640 1 -- 172.21.15.88:0/2661893178 <== mon.2 v1:172.21.15.88:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 344506374) 0x7f62240524b0 con 0x7f622c13f9a0 2024-03-29T10:45:08.369 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:08.386+0000 7f621affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:45:08.369 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:08.386+0000 7f621affd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:45:08.369 INFO:teuthology.misc.health.smithi088.stdout: 2024-03-29T10:45:08.370 INFO:teuthology.misc.health.smithi088.stdout:{"epoch":10,"fsid":"869f5188-f92b-42dd-a46d-8c646e0388a9","created":"2024-03-29T10:40:40.285087+0000","modified":"2024-03-29T10:40:42.536230+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:40:42.536230+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a26c0c1c-8de0-438c-a3f6-3df10ea61574","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5de24915-8dec-49b6-8b16-f3170197c591","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ac054af3-d589-4565-9644-e6af75975c53","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3ec4f045-c714-4a22-bd1a-e07330b45ab3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"cddb9867-8e30-4951-9b13-3d1e78b9c45a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dc6d9911-785d-432f-8b53-56a1ccab302f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"77c88437-b587-496e-8b4d-1824ea8b5305","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ce963d96-5a8e-44c3-99de-c0dbebe234ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:45:08.373 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:08.390+0000 7f622b59e640 10 librados: watch_flush enter 2024-03-29T10:45:08.373 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:08.390+0000 7f622b59e640 10 librados: watch_flush exit 2024-03-29T10:45:08.373 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:08.390+0000 7f622b59e640 20 client.4409.objecter shutdown clearing up homeless session... 2024-03-29T10:45:08.373 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:08.390+0000 7f622b59e640 10 client.4409.objecter successfully canceled tick 2024-03-29T10:45:08.373 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:08.390+0000 7f622b59e640 1 -- 172.21.15.88:0/2661893178 >> v1:172.21.15.88:6816/1752861349 conn(0x7f620c06ded0 legacy=0x7f620c070390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:08.374 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:08.390+0000 7f622b59e640 10 monclient: shutdown 2024-03-29T10:45:08.374 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:08.390+0000 7f622b59e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:45:08.374 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:08.390+0000 7f622b59e640 1 -- 172.21.15.88:0/2661893178 >> v1:172.21.15.88:6790/0 conn(0x7f622c13f9a0 legacy=0x7f622c0ac440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:08.374 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:08.390+0000 7f622b59e640 1 -- 172.21.15.88:0/2661893178 shutdown_connections 2024-03-29T10:45:08.374 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:08.390+0000 7f622b59e640 1 -- 172.21.15.88:0/2661893178 >> 172.21.15.88:0/2661893178 conn(0x7f622c06d230 msgr2=0x7f622c069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:45:08.374 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:08.391+0000 7f622b59e640 1 -- 172.21.15.88:0/2661893178 shutdown_connections 2024-03-29T10:45:08.374 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:08.391+0000 7f622b59e640 1 -- 172.21.15.88:0/2661893178 wait complete. 2024-03-29T10:45:08.374 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:08.391+0000 7f622b59e640 1 librados: shutdown 2024-03-29T10:45:08.388 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:45:15.196 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:45:15.349 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:15.367+0000 7f1359c9d640 10 monclient: get_monmap_and_config 2024-03-29T10:45:15.350 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:15.367+0000 7f1359c9d640 10 monclient: build_initial_monmap 2024-03-29T10:45:15.350 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:15.367+0000 7f1359c9d640 10 monclient: monmap: 2024-03-29T10:45:15.350 INFO:teuthology.misc.health.smithi088.stderr:epoch 0 2024-03-29T10:45:15.350 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:45:15.350 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:45:15.368211+0000 2024-03-29T10:45:15.350 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:45:15.368211+0000 2024-03-29T10:45:15.350 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 0 (unknown) 2024-03-29T10:45:15.350 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:45:15.350 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-03-29T10:45:15.350 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-29T10:45:15.350 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-b 2024-03-29T10:45:15.351 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:45:15.351 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:15.368+0000 7f1359c9d640 1 Processor -- start 2024-03-29T10:45:15.351 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:15.368+0000 7f1359c9d640 1 -- start start 2024-03-29T10:45:15.351 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:15.368+0000 7f1359c9d640 10 monclient: init 2024-03-29T10:45:15.352 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:15.369+0000 7f1359c9d640 10 monclient: _reopen_session rank -1 2024-03-29T10:45:15.352 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:15.369+0000 7f1359c9d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:45:15.352 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:15.369+0000 7f1359c9d640 10 monclient(hunting): picked mon.noname-b con 0x7f13541422b0 addr v1:172.21.15.88:6790/0 2024-03-29T10:45:15.352 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:15.369+0000 7f1359c9d640 10 monclient(hunting): picked mon.noname-c con 0x7f1354142690 addr v1:172.21.15.196:6789/0 2024-03-29T10:45:15.352 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:15.369+0000 7f1359c9d640 10 monclient(hunting): picked mon.noname-a con 0x7f13540b17d0 addr v1:172.21.15.88:6789/0 2024-03-29T10:45:15.352 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:15.369+0000 7f1359c9d640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1354142c80 con 0x7f13540b17d0 2024-03-29T10:45:15.352 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:15.369+0000 7f1359c9d640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13540aae40 con 0x7f1354142690 2024-03-29T10:45:15.352 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:15.369+0000 7f1359c9d640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f135413fbd0 con 0x7f13541422b0 2024-03-29T10:45:15.352 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:15.369+0000 7f1359c9d640 10 monclient(hunting): _renew_subs 2024-03-29T10:45:15.352 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:15.369+0000 7f1359c9d640 10 monclient(hunting): authenticate will time out at 1037.605263s 2024-03-29T10:45:15.353 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:15.369+0000 7f1352ffd640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f1354142690 0x7f13540b2590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.88:59246/0 (socket says 172.21.15.88:59246) 2024-03-29T10:45:15.353 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:15.369+0000 7f1352ffd640 1 -- 172.21.15.88:0/1221576677 learned_addr learned my addr 172.21.15.88:0/1221576677 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-29T10:45:15.353 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:15.370+0000 7f13527fc640 1 -- 172.21.15.88:0/1221576677 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1657182582 0 0) 0x7f13540aae40 con 0x7f1354142690 2024-03-29T10:45:15.353 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:15.370+0000 7f13527fc640 10 monclient(hunting): _negotiate 2024-03-29T10:45:15.354 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:15.370+0000 7f13527fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:45:15.354 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:15.370+0000 7f13527fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:45:15.354 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:15.370+0000 7f13527fc640 10 monclient(hunting): my global_id is 4384 2024-03-29T10:45:15.354 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:15.370+0000 7f13527fc640 1 -- 172.21.15.88:0/1221576677 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1348003520 con 0x7f1354142690 2024-03-29T10:45:15.354 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:15.370+0000 7f13527fc640 1 -- 172.21.15.88:0/1221576677 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3352819581 0 0) 0x7f1348003520 con 0x7f1354142690 2024-03-29T10:45:15.354 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:15.370+0000 7f13527fc640 1 -- 172.21.15.88:0/1221576677 >> v1:172.21.15.88:6790/0 conn(0x7f13541422b0 legacy=0x7f135413f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:15.354 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:15.370+0000 7f13527fc640 1 -- 172.21.15.88:0/1221576677 >> v1:172.21.15.88:6789/0 conn(0x7f13540b17d0 legacy=0x7f13540b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:15.354 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:15.370+0000 7f13527fc640 10 monclient: _finish_hunting 0 2024-03-29T10:45:15.354 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:15.370+0000 7f13527fc640 1 monclient: found mon.noname-c 2024-03-29T10:45:15.354 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:15.370+0000 7f13527fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:45:15.354 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:15.370+0000 7f13527fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.196:6789/0 2024-03-29T10:45:15.354 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:15.370+0000 7f13527fc640 1 -- 172.21.15.88:0/1221576677 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1354151940 con 0x7f1354142690 2024-03-29T10:45:15.354 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:15.370+0000 7f13527fc640 10 monclient: _finish_auth 0 2024-03-29T10:45:15.355 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:15.370+0000 7f13527fc640 10 monclient: _check_auth_tickets 2024-03-29T10:45:15.355 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:15.370+0000 7f13527fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:45:15.355 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:15.370+0000 7f13527fc640 1 -- 172.21.15.88:0/1221576677 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f133c001690 con 0x7f1354142690 2024-03-29T10:45:15.355 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:15.370+0000 7f13527fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:15.355 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:15.370+0000 7f13527fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:45:15.355 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:15.370+0000 7f13527fc640 10 monclient: dump: 2024-03-29T10:45:15.355 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:45:15.355 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:45:15.355 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:45:15.355 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:45:15.355 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:15.355 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:45:15.355 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:45:15.355 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:45:15.356 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:45:15.356 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:45:15.356 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:15.370+0000 7f1359c9d640 5 monclient: authenticate success, global_id 4384 2024-03-29T10:45:15.356 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:15.370+0000 7f1359c9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:45:15.360 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:15.371+0000 7f13527fc640 1 -- 172.21.15.88:0/1221576677 <== mon.1 v1:172.21.15.196:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f133c0017f0 con 0x7f1354142690 2024-03-29T10:45:15.361 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:15.371+0000 7f13527fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:45:15.361 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:15.371+0000 7f13527fc640 1 -- 172.21.15.88:0/1221576677 <== mon.1 v1:172.21.15.196:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f133c001bc0 con 0x7f1354142690 2024-03-29T10:45:15.361 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:15.371+0000 7f13527fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:15.361 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:15.371+0000 7f13527fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:45:15.361 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:15.371+0000 7f13527fc640 10 monclient: dump: 2024-03-29T10:45:15.361 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:45:15.361 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:45:15.361 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:45:15.361 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:45:15.361 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:15.361 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:45:15.361 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:45:15.361 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:45:15.362 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:45:15.362 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:45:15.362 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:15.371+0000 7f1359c9d640 10 monclient: get_monmap_and_config success 2024-03-29T10:45:15.362 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:15.371+0000 7f1359c9d640 4 set_mon_vals no callback set 2024-03-29T10:45:15.362 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:15.372+0000 7f1359c9d640 10 monclient: shutdown 2024-03-29T10:45:15.362 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:15.372+0000 7f1359c9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:45:15.362 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:15.372+0000 7f1359c9d640 1 -- 172.21.15.88:0/1221576677 >> v1:172.21.15.196:6789/0 conn(0x7f1354142690 legacy=0x7f13540b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:15.362 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:15.372+0000 7f1359c9d640 1 -- 172.21.15.88:0/1221576677 shutdown_connections 2024-03-29T10:45:15.362 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:15.372+0000 7f1359c9d640 1 -- 172.21.15.88:0/1221576677 >> 172.21.15.88:0/1221576677 conn(0x7f1354066a00 msgr2=0x7f1354066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:45:15.362 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:15.372+0000 7f1359c9d640 1 -- 172.21.15.88:0/1221576677 shutdown_connections 2024-03-29T10:45:15.362 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:15.372+0000 7f1359c9d640 1 -- 172.21.15.88:0/1221576677 wait complete. 2024-03-29T10:45:15.362 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:15.373+0000 7f1359c9d640 10 monclient: build_initial_monmap 2024-03-29T10:45:15.362 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:15.373+0000 7f1359c9d640 10 monclient: monmap: 2024-03-29T10:45:15.363 INFO:teuthology.misc.health.smithi088.stderr:epoch 0 2024-03-29T10:45:15.363 INFO:teuthology.misc.health.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:45:15.363 INFO:teuthology.misc.health.smithi088.stderr:last_changed 0.000000 2024-03-29T10:45:15.363 INFO:teuthology.misc.health.smithi088.stderr:created 0.000000 2024-03-29T10:45:15.363 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 0 (unknown) 2024-03-29T10:45:15.363 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:45:15.363 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-03-29T10:45:15.363 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-29T10:45:15.363 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-c 2024-03-29T10:45:15.363 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:45:15.363 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:15.373+0000 7f1359c9d640 1 librados: starting msgr at 2024-03-29T10:45:15.363 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:15.373+0000 7f1359c9d640 1 librados: starting objecter 2024-03-29T10:45:15.363 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:15.374+0000 7f1359c9d640 1 Processor -- start 2024-03-29T10:45:15.363 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:15.374+0000 7f1359c9d640 1 -- start start 2024-03-29T10:45:15.364 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:15.374+0000 7f1359c9d640 1 librados: setting wanted keys 2024-03-29T10:45:15.364 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:15.374+0000 7f1359c9d640 1 librados: calling monclient init 2024-03-29T10:45:15.364 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:15.374+0000 7f1359c9d640 10 monclient: init 2024-03-29T10:45:15.364 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:15.375+0000 7f1359c9d640 10 monclient: _reopen_session rank -1 2024-03-29T10:45:15.364 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:15.375+0000 7f1359c9d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:45:15.364 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:15.375+0000 7f1359c9d640 10 monclient(hunting): picked mon.noname-c con 0x7f13540b17d0 addr v1:172.21.15.88:6790/0 2024-03-29T10:45:15.364 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:15.375+0000 7f1359c9d640 10 monclient(hunting): picked mon.noname-b con 0x7f13541422b0 addr v1:172.21.15.196:6789/0 2024-03-29T10:45:15.364 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:15.375+0000 7f1359c9d640 10 monclient(hunting): picked mon.noname-a con 0x7f1354142690 addr v1:172.21.15.88:6789/0 2024-03-29T10:45:15.364 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:15.375+0000 7f1359c9d640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f135406f3b0 con 0x7f1354142690 2024-03-29T10:45:15.364 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:15.375+0000 7f1359c9d640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13540b7760 con 0x7f13541422b0 2024-03-29T10:45:15.364 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:15.375+0000 7f1359c9d640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f135413c980 con 0x7f13540b17d0 2024-03-29T10:45:15.364 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:15.375+0000 7f1359c9d640 10 monclient(hunting): _renew_subs 2024-03-29T10:45:15.364 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:15.375+0000 7f1359c9d640 10 monclient(hunting): authenticate will time out at 1037.610980s 2024-03-29T10:45:15.364 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:15.375+0000 7f13537fe640 1 --1- >> v1:172.21.15.88:6790/0 conn(0x7f13540b17d0 0x7f13540b59a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6790/0 says I am v1:172.21.15.88:38754/0 (socket says 172.21.15.88:38754) 2024-03-29T10:45:15.365 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:15.375+0000 7f13537fe640 1 -- 172.21.15.88:0/2360755394 learned_addr learned my addr 172.21.15.88:0/2360755394 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-29T10:45:15.365 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:15.375+0000 7f1343fff640 10 client.?.objecter ms_handle_connect 0x7f13540b17d0 2024-03-29T10:45:15.365 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:15.375+0000 7f1343fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:45:15.365 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:15.375+0000 7f1343fff640 10 client.?.objecter ms_handle_connect 0x7f1354142690 2024-03-29T10:45:15.365 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:15.375+0000 7f1343fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:45:15.365 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:15.375+0000 7f1343fff640 10 client.?.objecter ms_handle_connect 0x7f13541422b0 2024-03-29T10:45:15.365 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:15.375+0000 7f1343fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:45:15.365 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:15.375+0000 7f1343fff640 1 -- 172.21.15.88:0/2360755394 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2533345814 0 0) 0x7f135406f3b0 con 0x7f1354142690 2024-03-29T10:45:15.365 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:15.376+0000 7f1343fff640 10 monclient(hunting): _negotiate 2024-03-29T10:45:15.365 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:15.376+0000 7f1343fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:45:15.365 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:15.376+0000 7f1343fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:45:15.365 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:15.376+0000 7f1343fff640 10 monclient(hunting): my global_id is 4386 2024-03-29T10:45:15.365 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:15.376+0000 7f1343fff640 1 -- 172.21.15.88:0/2360755394 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1334004eb0 con 0x7f1354142690 2024-03-29T10:45:15.365 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:15.376+0000 7f1343fff640 1 -- 172.21.15.88:0/2360755394 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3257063167 0 0) 0x7f13540b7760 con 0x7f13541422b0 2024-03-29T10:45:15.366 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:15.376+0000 7f1343fff640 10 monclient(hunting): _negotiate 2024-03-29T10:45:15.366 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:15.376+0000 7f1343fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:45:15.366 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:15.376+0000 7f1343fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:45:15.366 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:15.376+0000 7f1343fff640 10 monclient(hunting): my global_id is 4387 2024-03-29T10:45:15.366 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:15.376+0000 7f1343fff640 1 -- 172.21.15.88:0/2360755394 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f135406f3b0 con 0x7f13541422b0 2024-03-29T10:45:15.366 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:15.376+0000 7f1343fff640 1 -- 172.21.15.88:0/2360755394 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 876112333 0 0) 0x7f135413c980 con 0x7f13540b17d0 2024-03-29T10:45:15.366 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:15.376+0000 7f1343fff640 10 monclient(hunting): _negotiate 2024-03-29T10:45:15.366 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:15.376+0000 7f1343fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:45:15.366 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:15.376+0000 7f1343fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:45:15.366 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:15.376+0000 7f1343fff640 10 monclient(hunting): my global_id is 4415 2024-03-29T10:45:15.366 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:15.376+0000 7f1343fff640 1 -- 172.21.15.88:0/2360755394 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f13540b7760 con 0x7f13540b17d0 2024-03-29T10:45:15.367 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:15.376+0000 7f1343fff640 1 -- 172.21.15.88:0/2360755394 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1240274333 0 0) 0x7f1334004eb0 con 0x7f1354142690 2024-03-29T10:45:15.367 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:15.376+0000 7f1343fff640 1 -- 172.21.15.88:0/2360755394 --> v1:172.21.15.88:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f135413c980 con 0x7f1354142690 2024-03-29T10:45:15.367 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:15.376+0000 7f1343fff640 1 -- 172.21.15.88:0/2360755394 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f134c001bc0 con 0x7f1354142690 2024-03-29T10:45:15.367 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:15.376+0000 7f1343fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:15.367 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:15.376+0000 7f1343fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:45:15.367 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:15.376+0000 7f1343fff640 10 monclient(hunting): dump: 2024-03-29T10:45:15.367 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:45:15.367 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:45:15.367 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:45:15.367 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:45:15.367 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:15.367 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:45:15.367 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:45:15.367 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:45:15.368 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:45:15.368 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:45:15.368 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:15.376+0000 7f1343fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:45:15.368 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:15.376+0000 7f1343fff640 1 -- 172.21.15.88:0/2360755394 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1623711185 0 0) 0x7f13540b7760 con 0x7f13540b17d0 2024-03-29T10:45:15.368 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:15.376+0000 7f1343fff640 1 -- 172.21.15.88:0/2360755394 --> v1:172.21.15.88:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1334004eb0 con 0x7f13540b17d0 2024-03-29T10:45:15.368 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:15.376+0000 7f1343fff640 1 -- 172.21.15.88:0/2360755394 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f1344001970 con 0x7f13540b17d0 2024-03-29T10:45:15.368 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:15.376+0000 7f1343fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:15.368 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:15.376+0000 7f1343fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:45:15.368 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:15.376+0000 7f1343fff640 10 monclient(hunting): dump: 2024-03-29T10:45:15.368 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:45:15.368 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:45:15.368 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:45:15.368 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:45:15.368 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:15.369 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:45:15.369 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:45:15.369 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:45:15.369 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:45:15.369 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:45:15.369 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:15.376+0000 7f1343fff640 1 -- 172.21.15.88:0/2360755394 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2729554087 0 0) 0x7f135406f3b0 con 0x7f13541422b0 2024-03-29T10:45:15.369 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:15.376+0000 7f1343fff640 1 -- 172.21.15.88:0/2360755394 --> v1:172.21.15.196:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f13540b7760 con 0x7f13541422b0 2024-03-29T10:45:15.369 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:15.376+0000 7f1343fff640 1 -- 172.21.15.88:0/2360755394 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f133c001b00 con 0x7f13541422b0 2024-03-29T10:45:15.369 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:15.376+0000 7f1343fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:15.369 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:15.376+0000 7f1343fff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T10:45:15.370 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:15.376+0000 7f1343fff640 10 monclient(hunting): dump: 2024-03-29T10:45:15.370 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:45:15.370 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:45:15.370 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:45:15.370 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:45:15.370 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:15.370 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:45:15.370 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:45:15.370 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:45:15.370 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:45:15.370 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:45:15.370 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:15.377+0000 7f1343fff640 1 -- 172.21.15.88:0/2360755394 <== mon.2 v1:172.21.15.88:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4270571679 0 0) 0x7f1334004eb0 con 0x7f13540b17d0 2024-03-29T10:45:15.370 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:15.377+0000 7f1343fff640 1 -- 172.21.15.88:0/2360755394 >> v1:172.21.15.196:6789/0 conn(0x7f13541422b0 legacy=0x7f13540b60b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:15.371 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:15.377+0000 7f1343fff640 1 -- 172.21.15.88:0/2360755394 >> v1:172.21.15.88:6789/0 conn(0x7f1354142690 legacy=0x7f13541e6450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:15.371 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:15.377+0000 7f1343fff640 10 monclient: _finish_hunting 0 2024-03-29T10:45:15.371 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:15.377+0000 7f1343fff640 1 monclient: found mon.c 2024-03-29T10:45:15.371 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:15.377+0000 7f1343fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:45:15.371 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:15.377+0000 7f1343fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-03-29T10:45:15.371 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:15.377+0000 7f1343fff640 1 -- 172.21.15.88:0/2360755394 --> v1:172.21.15.88:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13540b6bd0 con 0x7f13540b17d0 2024-03-29T10:45:15.371 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:15.377+0000 7f1343fff640 10 monclient: _finish_auth 0 2024-03-29T10:45:15.371 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:15.377+0000 7f1343fff640 10 monclient: _check_auth_tickets 2024-03-29T10:45:15.371 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:15.377+0000 7f1343fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:45:15.371 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:15.377+0000 7f1359c9d640 5 monclient: authenticate success, global_id 4415 2024-03-29T10:45:15.371 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:15.377+0000 7f1359c9d640 10 monclient: _renew_subs 2024-03-29T10:45:15.371 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:15.377+0000 7f1359c9d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-03-29T10:45:15.371 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:15.377+0000 7f1359c9d640 1 -- 172.21.15.88:0/2360755394 --> v1:172.21.15.88:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f135406e380 con 0x7f13540b17d0 2024-03-29T10:45:15.371 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:15.377+0000 7f1359c9d640 10 client.4415.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:45:15.372 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:15.377+0000 7f1359c9d640 10 monclient: _renew_subs 2024-03-29T10:45:15.372 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:15.377+0000 7f1359c9d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-03-29T10:45:15.372 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:15.377+0000 7f1359c9d640 1 -- 172.21.15.88:0/2360755394 --> v1:172.21.15.88:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1354150710 con 0x7f13540b17d0 2024-03-29T10:45:15.372 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:15.377+0000 7f1359c9d640 1 librados: init done 2024-03-29T10:45:15.373 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:15.378+0000 7f1343fff640 1 -- 172.21.15.88:0/2360755394 <== mon.2 v1:172.21.15.88:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1344003610 con 0x7f13540b17d0 2024-03-29T10:45:15.373 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:15.378+0000 7f1343fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:45:15.373 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:15.378+0000 7f1343fff640 1 -- 172.21.15.88:0/2360755394 <== mon.2 v1:172.21.15.88:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f1344003960 con 0x7f13540b17d0 2024-03-29T10:45:15.373 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:15.378+0000 7f1343fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:15.373 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:15.378+0000 7f1343fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:45:15.373 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:15.378+0000 7f1343fff640 10 monclient: dump: 2024-03-29T10:45:15.373 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:45:15.373 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:45:15.373 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:45:15.373 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:45:15.373 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:15.373 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:45:15.373 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:45:15.374 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:45:15.374 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:45:15.374 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:45:15.374 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:15.378+0000 7f1351ffb640 4 set_mon_vals no callback set 2024-03-29T10:45:15.374 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:15.380+0000 7f1343fff640 1 -- 172.21.15.88:0/2360755394 <== mon.2 v1:172.21.15.88:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4235629220 0 0) 0x7f1344003ac0 con 0x7f13540b17d0 2024-03-29T10:45:15.374 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:15.381+0000 7f1343fff640 1 -- 172.21.15.88:0/2360755394 <== mon.2 v1:172.21.15.88:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4257673582 0 0) 0x7f1344084a20 con 0x7f13540b17d0 2024-03-29T10:45:15.374 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:15.381+0000 7f1343fff640 10 client.4415.objecter ms_dispatch 0x7f1354000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:45:15.374 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:15.381+0000 7f1343fff640 3 client.4415.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:45:15.374 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:15.381+0000 7f1343fff640 3 client.4415.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:45:15.374 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:15.381+0000 7f1343fff640 20 client.4415.objecter dump_active .. 0 homeless 2024-03-29T10:45:15.374 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:15.381+0000 7f1352ffd640 10 monclient: get_auth_request con 0x7f133406e100 auth_method 0 2024-03-29T10:45:15.374 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:15.381+0000 7f1359c9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:45:15.374 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:15.381+0000 7f1359c9d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:45:15.375 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:15.381+0000 7f1359c9d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-03-29T10:45:15.375 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:15.381+0000 7f1359c9d640 1 -- 172.21.15.88:0/2360755394 --> v1:172.21.15.88:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1338005080 con 0x7f13540b17d0 2024-03-29T10:45:15.375 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:15.382+0000 7f1343fff640 10 client.4415.objecter ms_handle_connect 0x7f133406e100 2024-03-29T10:45:15.375 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:15.391+0000 7f1343fff640 1 -- 172.21.15.88:0/2360755394 <== mon.2 v1:172.21.15.88:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1344050300 con 0x7f13540b17d0 2024-03-29T10:45:15.375 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:15.391+0000 7f1343fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:45:15.375 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:15.391+0000 7f1343fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:45:15.746 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:15.762+0000 7f1359c9d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:45:15.746 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:15.762+0000 7f1359c9d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:45:15.746 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:15.762+0000 7f1359c9d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-03-29T10:45:15.746 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:15.762+0000 7f1359c9d640 1 -- 172.21.15.88:0/2360755394 --> v1:172.21.15.88:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1338005330 con 0x7f13540b17d0 2024-03-29T10:45:15.747 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:15.763+0000 7f1343fff640 1 -- 172.21.15.88:0/2360755394 <== mon.2 v1:172.21.15.88:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 344506374) 0x7f1344052230 con 0x7f13540b17d0 2024-03-29T10:45:15.747 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:15.763+0000 7f1343fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:45:15.747 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:15.763+0000 7f1343fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:45:15.747 INFO:teuthology.misc.health.smithi088.stdout: 2024-03-29T10:45:15.747 INFO:teuthology.misc.health.smithi088.stdout:{"epoch":10,"fsid":"869f5188-f92b-42dd-a46d-8c646e0388a9","created":"2024-03-29T10:40:40.285087+0000","modified":"2024-03-29T10:40:42.536230+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:40:42.536230+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a26c0c1c-8de0-438c-a3f6-3df10ea61574","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5de24915-8dec-49b6-8b16-f3170197c591","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ac054af3-d589-4565-9644-e6af75975c53","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3ec4f045-c714-4a22-bd1a-e07330b45ab3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"cddb9867-8e30-4951-9b13-3d1e78b9c45a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dc6d9911-785d-432f-8b53-56a1ccab302f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"77c88437-b587-496e-8b4d-1824ea8b5305","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ce963d96-5a8e-44c3-99de-c0dbebe234ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:45:15.750 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:15.767+0000 7f1359c9d640 10 librados: watch_flush enter 2024-03-29T10:45:15.750 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:15.767+0000 7f1359c9d640 10 librados: watch_flush exit 2024-03-29T10:45:15.750 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:15.767+0000 7f1359c9d640 20 client.4415.objecter shutdown clearing up homeless session... 2024-03-29T10:45:15.750 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:15.767+0000 7f1359c9d640 10 client.4415.objecter successfully canceled tick 2024-03-29T10:45:15.750 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:15.767+0000 7f1359c9d640 1 -- 172.21.15.88:0/2360755394 >> v1:172.21.15.88:6816/1752861349 conn(0x7f133406e100 legacy=0x7f13340705c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:15.750 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:15.767+0000 7f1359c9d640 10 monclient: shutdown 2024-03-29T10:45:15.750 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:15.767+0000 7f1359c9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:45:15.750 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:15.767+0000 7f1359c9d640 1 -- 172.21.15.88:0/2360755394 >> v1:172.21.15.88:6790/0 conn(0x7f13540b17d0 legacy=0x7f13540b59a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:15.751 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:15.768+0000 7f1359c9d640 1 -- 172.21.15.88:0/2360755394 shutdown_connections 2024-03-29T10:45:15.751 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:15.768+0000 7f1359c9d640 1 -- 172.21.15.88:0/2360755394 >> 172.21.15.88:0/2360755394 conn(0x7f1354066a00 msgr2=0x7f135413e710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:45:15.751 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:15.768+0000 7f1359c9d640 1 -- 172.21.15.88:0/2360755394 shutdown_connections 2024-03-29T10:45:15.751 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:15.768+0000 7f1359c9d640 1 -- 172.21.15.88:0/2360755394 wait complete. 2024-03-29T10:45:15.751 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:15.768+0000 7f1359c9d640 1 librados: shutdown 2024-03-29T10:45:15.767 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:45:22.575 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:45:22.731 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:22.748+0000 7f57725d6640 10 monclient: get_monmap_and_config 2024-03-29T10:45:22.731 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:22.748+0000 7f57725d6640 10 monclient: build_initial_monmap 2024-03-29T10:45:22.732 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:22.748+0000 7f57725d6640 10 monclient: monmap: 2024-03-29T10:45:22.732 INFO:teuthology.misc.health.smithi088.stderr:epoch 0 2024-03-29T10:45:22.732 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:45:22.732 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:45:22.749291+0000 2024-03-29T10:45:22.732 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:45:22.749291+0000 2024-03-29T10:45:22.732 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 0 (unknown) 2024-03-29T10:45:22.732 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:45:22.732 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-03-29T10:45:22.732 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-29T10:45:22.732 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-b 2024-03-29T10:45:22.732 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:45:22.733 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:22.751+0000 7f57725d6640 1 Processor -- start 2024-03-29T10:45:22.736 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:22.751+0000 7f57725d6640 1 -- start start 2024-03-29T10:45:22.736 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:22.751+0000 7f57725d6640 10 monclient: init 2024-03-29T10:45:22.736 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:22.752+0000 7f57725d6640 10 monclient: _reopen_session rank -1 2024-03-29T10:45:22.736 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:22.752+0000 7f57725d6640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:45:22.738 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:22.754+0000 7f57725d6640 10 monclient(hunting): picked mon.noname-b con 0x7f576c0a9ac0 addr v1:172.21.15.88:6790/0 2024-03-29T10:45:22.739 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:22.754+0000 7f57725d6640 10 monclient(hunting): picked mon.noname-c con 0x7f576c13ee90 addr v1:172.21.15.196:6789/0 2024-03-29T10:45:22.740 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:22.754+0000 7f57725d6640 10 monclient(hunting): picked mon.noname-a con 0x7f576c0b4100 addr v1:172.21.15.88:6789/0 2024-03-29T10:45:22.740 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:22.754+0000 7f57725d6640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f576c0af6f0 con 0x7f576c0b4100 2024-03-29T10:45:22.740 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:22.754+0000 7f57725d6640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f576c0aac80 con 0x7f576c13ee90 2024-03-29T10:45:22.740 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:22.754+0000 7f57725d6640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f576c0aaf60 con 0x7f576c0a9ac0 2024-03-29T10:45:22.740 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:22.754+0000 7f57725d6640 10 monclient(hunting): _renew_subs 2024-03-29T10:45:22.740 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:22.754+0000 7f57725d6640 10 monclient(hunting): authenticate will time out at 1044.990826s 2024-03-29T10:45:22.742 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:22.756+0000 7f576bfff640 1 --1- >> v1:172.21.15.88:6790/0 conn(0x7f576c0a9ac0 0x7f576c0adfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6790/0 says I am v1:172.21.15.88:56315/0 (socket says 172.21.15.88:56315) 2024-03-29T10:45:22.742 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:22.756+0000 7f576bfff640 1 -- 172.21.15.88:0/1158481551 learned_addr learned my addr 172.21.15.88:0/1158481551 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-29T10:45:22.742 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:22.756+0000 7f576affd640 1 -- 172.21.15.88:0/1158481551 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3719275857 0 0) 0x7f576c0aaf60 con 0x7f576c0a9ac0 2024-03-29T10:45:22.742 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:22.756+0000 7f576affd640 10 monclient(hunting): _negotiate 2024-03-29T10:45:22.742 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:22.756+0000 7f576affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:45:22.742 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:22.756+0000 7f576affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:45:22.742 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:22.756+0000 7f576affd640 10 monclient(hunting): my global_id is 4418 2024-03-29T10:45:22.742 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:22.756+0000 7f576affd640 1 -- 172.21.15.88:0/1158481551 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5754003520 con 0x7f576c0a9ac0 2024-03-29T10:45:22.742 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:22.758+0000 7f576affd640 1 -- 172.21.15.88:0/1158481551 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4132660697 0 0) 0x7f5754003520 con 0x7f576c0a9ac0 2024-03-29T10:45:22.742 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:22.758+0000 7f576affd640 1 -- 172.21.15.88:0/1158481551 >> v1:172.21.15.196:6789/0 conn(0x7f576c13ee90 legacy=0x7f576c13b5f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:45:22.742 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:22.758+0000 7f576affd640 1 -- 172.21.15.88:0/1158481551 >> v1:172.21.15.88:6789/0 conn(0x7f576c0b4100 legacy=0x7f576c14a390 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:45:22.744 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:22.760+0000 7f576affd640 10 monclient: _finish_hunting 0 2024-03-29T10:45:22.745 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:22.760+0000 7f576affd640 1 monclient: found mon.noname-b 2024-03-29T10:45:22.745 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:22.760+0000 7f576affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:45:22.745 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:22.760+0000 7f576affd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.88:6790/0 2024-03-29T10:45:22.745 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:22.760+0000 7f576affd640 1 -- 172.21.15.88:0/1158481551 --> v1:172.21.15.88:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f576c13e970 con 0x7f576c0a9ac0 2024-03-29T10:45:22.745 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:22.761+0000 7f576affd640 10 monclient: _finish_auth 0 2024-03-29T10:45:22.745 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:22.761+0000 7f576affd640 10 monclient: _check_auth_tickets 2024-03-29T10:45:22.746 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:22.761+0000 7f576affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:45:22.746 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:22.761+0000 7f576affd640 1 -- 172.21.15.88:0/1158481551 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f575c001690 con 0x7f576c0a9ac0 2024-03-29T10:45:22.746 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:22.761+0000 7f576affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:22.746 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:22.761+0000 7f576affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:45:22.746 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:22.761+0000 7f576affd640 10 monclient: dump: 2024-03-29T10:45:22.746 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:45:22.746 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:45:22.747 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:45:22.747 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:45:22.747 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:22.747 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:45:22.747 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:45:22.747 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:45:22.748 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:45:22.748 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:45:22.749 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:22.762+0000 7f57725d6640 5 monclient: authenticate success, global_id 4418 2024-03-29T10:45:22.749 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:22.762+0000 7f57725d6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:45:22.750 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:22.763+0000 7f576affd640 1 -- 172.21.15.88:0/1158481551 <== mon.2 v1:172.21.15.88:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f575c001860 con 0x7f576c0a9ac0 2024-03-29T10:45:22.750 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:22.763+0000 7f576affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:45:22.750 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:22.763+0000 7f576affd640 1 -- 172.21.15.88:0/1158481551 <== mon.2 v1:172.21.15.88:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f575c001cb0 con 0x7f576c0a9ac0 2024-03-29T10:45:22.750 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:22.763+0000 7f576affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:22.750 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:22.763+0000 7f576affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:45:22.751 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:22.763+0000 7f576affd640 10 monclient: dump: 2024-03-29T10:45:22.751 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:45:22.751 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:45:22.751 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:45:22.751 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:45:22.751 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:22.752 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:45:22.752 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:45:22.752 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:45:22.752 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:45:22.752 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:45:22.752 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:22.763+0000 7f57725d6640 10 monclient: get_monmap_and_config success 2024-03-29T10:45:22.752 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:22.763+0000 7f57725d6640 4 set_mon_vals no callback set 2024-03-29T10:45:22.753 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:22.764+0000 7f57725d6640 10 monclient: shutdown 2024-03-29T10:45:22.753 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:22.764+0000 7f57725d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:45:22.753 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:22.764+0000 7f57725d6640 1 -- 172.21.15.88:0/1158481551 >> v1:172.21.15.88:6790/0 conn(0x7f576c0a9ac0 legacy=0x7f576c0adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:22.756 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:22.768+0000 7f57725d6640 1 -- 172.21.15.88:0/1158481551 shutdown_connections 2024-03-29T10:45:22.756 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:22.768+0000 7f57725d6640 1 -- 172.21.15.88:0/1158481551 >> 172.21.15.88:0/1158481551 conn(0x7f576c0b35e0 msgr2=0x7f576c0b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:45:22.756 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:22.769+0000 7f57725d6640 1 -- 172.21.15.88:0/1158481551 shutdown_connections 2024-03-29T10:45:22.756 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:22.769+0000 7f57725d6640 1 -- 172.21.15.88:0/1158481551 wait complete. 2024-03-29T10:45:22.756 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:22.770+0000 7f57725d6640 10 monclient: build_initial_monmap 2024-03-29T10:45:22.756 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:22.770+0000 7f57725d6640 10 monclient: monmap: 2024-03-29T10:45:22.756 INFO:teuthology.misc.health.smithi088.stderr:epoch 0 2024-03-29T10:45:22.756 INFO:teuthology.misc.health.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:45:22.757 INFO:teuthology.misc.health.smithi088.stderr:last_changed 0.000000 2024-03-29T10:45:22.757 INFO:teuthology.misc.health.smithi088.stderr:created 0.000000 2024-03-29T10:45:22.757 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 0 (unknown) 2024-03-29T10:45:22.757 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:45:22.757 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-03-29T10:45:22.757 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-29T10:45:22.757 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-c 2024-03-29T10:45:22.757 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:45:22.757 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:22.771+0000 7f57725d6640 1 librados: starting msgr at 2024-03-29T10:45:22.757 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:22.771+0000 7f57725d6640 1 librados: starting objecter 2024-03-29T10:45:22.758 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:22.771+0000 7f57725d6640 1 Processor -- start 2024-03-29T10:45:22.758 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:22.771+0000 7f57725d6640 1 -- start start 2024-03-29T10:45:22.758 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:22.771+0000 7f57725d6640 1 librados: setting wanted keys 2024-03-29T10:45:22.758 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:22.771+0000 7f57725d6640 1 librados: calling monclient init 2024-03-29T10:45:22.758 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:22.771+0000 7f57725d6640 10 monclient: init 2024-03-29T10:45:22.760 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:22.772+0000 7f57725d6640 10 monclient: _reopen_session rank -1 2024-03-29T10:45:22.760 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:22.772+0000 7f57725d6640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:45:22.760 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:22.775+0000 7f57725d6640 10 monclient(hunting): picked mon.noname-a con 0x7f576c0a9ac0 addr v1:172.21.15.88:6789/0 2024-03-29T10:45:22.761 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:22.775+0000 7f57725d6640 10 monclient(hunting): picked mon.noname-c con 0x7f576c0b4100 addr v1:172.21.15.88:6790/0 2024-03-29T10:45:22.761 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:22.775+0000 7f57725d6640 10 monclient(hunting): picked mon.noname-b con 0x7f576c13ee90 addr v1:172.21.15.196:6789/0 2024-03-29T10:45:22.761 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:22.775+0000 7f57725d6640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f576c141e10 con 0x7f576c0a9ac0 2024-03-29T10:45:22.761 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:22.775+0000 7f57725d6640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f576c06f990 con 0x7f576c13ee90 2024-03-29T10:45:22.761 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:22.775+0000 7f57725d6640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f576c06fc70 con 0x7f576c0b4100 2024-03-29T10:45:22.761 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:22.775+0000 7f57725d6640 10 monclient(hunting): _renew_subs 2024-03-29T10:45:22.761 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:22.775+0000 7f57725d6640 10 monclient(hunting): authenticate will time out at 1045.011606s 2024-03-29T10:45:22.761 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:22.776+0000 7f576bfff640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f576c0a9ac0 0x7f576c070370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:43831/0 (socket says 172.21.15.88:43831) 2024-03-29T10:45:22.761 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:22.776+0000 7f576bfff640 1 -- 172.21.15.88:0/974314526 learned_addr learned my addr 172.21.15.88:0/974314526 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-29T10:45:22.762 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:22.777+0000 7f5768ff9640 10 client.?.objecter ms_handle_connect 0x7f576c0a9ac0 2024-03-29T10:45:22.762 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:22.777+0000 7f5768ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:45:22.763 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:22.778+0000 7f5768ff9640 1 -- 172.21.15.88:0/974314526 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 481236572 0 0) 0x7f576c141e10 con 0x7f576c0a9ac0 2024-03-29T10:45:22.763 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:22.778+0000 7f5768ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:45:22.763 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:22.778+0000 7f5768ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:45:22.763 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:22.778+0000 7f5768ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:45:22.763 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:22.778+0000 7f5768ff9640 10 monclient(hunting): my global_id is 4389 2024-03-29T10:45:22.763 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:22.778+0000 7f5768ff9640 1 -- 172.21.15.88:0/974314526 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5758004eb0 con 0x7f576c0a9ac0 2024-03-29T10:45:22.764 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:22.778+0000 7f5768ff9640 1 -- 172.21.15.88:0/974314526 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 563398418 0 0) 0x7f5758004eb0 con 0x7f576c0a9ac0 2024-03-29T10:45:22.764 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:22.778+0000 7f5768ff9640 1 -- 172.21.15.88:0/974314526 --> v1:172.21.15.88:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f576c141e10 con 0x7f576c0a9ac0 2024-03-29T10:45:22.764 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:22.778+0000 7f5768ff9640 1 -- 172.21.15.88:0/974314526 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f575c001b70 con 0x7f576c0a9ac0 2024-03-29T10:45:22.764 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:22.778+0000 7f5768ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:22.764 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:22.778+0000 7f5768ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:45:22.764 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:22.778+0000 7f5768ff9640 10 monclient(hunting): dump: 2024-03-29T10:45:22.764 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:45:22.764 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:45:22.764 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:45:22.765 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:45:22.765 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:22.765 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:45:22.765 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:45:22.765 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:45:22.765 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:45:22.765 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:45:22.765 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:22.778+0000 7f5768ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:45:22.765 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:22.779+0000 7f5768ff9640 10 client.?.objecter ms_handle_connect 0x7f576c13ee90 2024-03-29T10:45:22.765 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:22.779+0000 7f5768ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:45:22.765 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:22.779+0000 7f5768ff9640 1 -- 172.21.15.88:0/974314526 <== mon.0 v1:172.21.15.88:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 168855320 0 0) 0x7f576c141e10 con 0x7f576c0a9ac0 2024-03-29T10:45:22.766 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:22.779+0000 7f5768ff9640 1 -- 172.21.15.88:0/974314526 >> v1:172.21.15.88:6790/0 conn(0x7f576c0b4100 legacy=0x7f576c1249e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:45:22.766 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:22.779+0000 7f5768ff9640 1 -- 172.21.15.88:0/974314526 >> v1:172.21.15.196:6789/0 conn(0x7f576c13ee90 legacy=0x7f576c1dee00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:22.766 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:22.779+0000 7f5768ff9640 10 monclient: _finish_hunting 0 2024-03-29T10:45:22.766 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:22.779+0000 7f5768ff9640 1 monclient: found mon.a 2024-03-29T10:45:22.766 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:22.779+0000 7f5768ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:45:22.771 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:22.779+0000 7f5768ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-03-29T10:45:22.771 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:22.779+0000 7f5768ff9640 1 -- 172.21.15.88:0/974314526 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f576c142060 con 0x7f576c0a9ac0 2024-03-29T10:45:22.772 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:22.779+0000 7f5768ff9640 10 monclient: _finish_auth 0 2024-03-29T10:45:22.772 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:22.779+0000 7f5768ff9640 10 monclient: _check_auth_tickets 2024-03-29T10:45:22.772 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:22.779+0000 7f5768ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:45:22.772 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:22.779+0000 7f57725d6640 5 monclient: authenticate success, global_id 4389 2024-03-29T10:45:22.772 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:22.779+0000 7f57725d6640 10 monclient: _renew_subs 2024-03-29T10:45:22.773 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:22.779+0000 7f57725d6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-03-29T10:45:22.773 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:22.779+0000 7f57725d6640 1 -- 172.21.15.88:0/974314526 --> v1:172.21.15.88:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f576c070a80 con 0x7f576c0a9ac0 2024-03-29T10:45:22.773 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:22.780+0000 7f57725d6640 10 client.4389.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:45:22.773 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:22.780+0000 7f57725d6640 10 monclient: _renew_subs 2024-03-29T10:45:22.773 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:22.780+0000 7f57725d6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-03-29T10:45:22.773 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:22.780+0000 7f57725d6640 1 -- 172.21.15.88:0/974314526 --> v1:172.21.15.88:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f576c142540 con 0x7f576c0a9ac0 2024-03-29T10:45:22.774 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:22.780+0000 7f57725d6640 1 librados: init done 2024-03-29T10:45:22.774 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:22.781+0000 7f57527fc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:45:22.774 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:22.781+0000 7f57527fc640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:45:22.774 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:22.781+0000 7f57527fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-03-29T10:45:22.774 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:22.781+0000 7f57527fc640 1 -- 172.21.15.88:0/974314526 --> v1:172.21.15.88:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f576c075f10 con 0x7f576c0a9ac0 2024-03-29T10:45:22.774 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:22.782+0000 7f5768ff9640 1 -- 172.21.15.88:0/974314526 <== mon.0 v1:172.21.15.88:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f575c004bf0 con 0x7f576c0a9ac0 2024-03-29T10:45:22.775 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:22.782+0000 7f5768ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:45:22.775 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:22.782+0000 7f5769ffb640 4 set_mon_vals no callback set 2024-03-29T10:45:22.775 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:22.783+0000 7f5768ff9640 1 -- 172.21.15.88:0/974314526 <== mon.0 v1:172.21.15.88:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f575c005040 con 0x7f576c0a9ac0 2024-03-29T10:45:22.775 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:22.783+0000 7f5768ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:22.775 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:22.783+0000 7f5768ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:45:22.775 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:22.783+0000 7f5768ff9640 10 monclient: dump: 2024-03-29T10:45:22.776 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:45:22.776 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:45:22.776 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:45:22.776 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:45:22.776 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:22.776 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:45:22.777 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:45:22.777 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:45:22.777 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:45:22.777 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:45:22.777 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:22.785+0000 7f5768ff9640 1 -- 172.21.15.88:0/974314526 <== mon.0 v1:172.21.15.88:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4235629220 0 0) 0x7f575c01b010 con 0x7f576c0a9ac0 2024-03-29T10:45:22.777 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:22.786+0000 7f5768ff9640 1 -- 172.21.15.88:0/974314526 <== mon.0 v1:172.21.15.88:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4257673582 0 0) 0x7f575c084cc0 con 0x7f576c0a9ac0 2024-03-29T10:45:22.777 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:22.786+0000 7f5768ff9640 10 client.4389.objecter ms_dispatch 0x7f576c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:45:22.778 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:22.786+0000 7f5768ff9640 3 client.4389.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:45:22.778 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:22.786+0000 7f5768ff9640 3 client.4389.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:45:22.778 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:22.786+0000 7f5768ff9640 20 client.4389.objecter dump_active .. 0 homeless 2024-03-29T10:45:22.778 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:22.788+0000 7f576b7fe640 10 monclient: get_auth_request con 0x7f575806dc10 auth_method 0 2024-03-29T10:45:22.778 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:22.789+0000 7f5768ff9640 10 client.4389.objecter ms_handle_connect 0x7f575806dc10 2024-03-29T10:45:22.779 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:22.792+0000 7f5768ff9640 1 -- 172.21.15.88:0/974314526 <== mon.0 v1:172.21.15.88:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f575c0505a0 con 0x7f576c0a9ac0 2024-03-29T10:45:22.779 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:22.792+0000 7f5768ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:45:22.780 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:22.792+0000 7f5768ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:45:23.100 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:23.116+0000 7f57527fc640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:45:23.101 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:23.116+0000 7f57527fc640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:45:23.101 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:23.116+0000 7f57527fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-03-29T10:45:23.101 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:23.116+0000 7f57527fc640 1 -- 172.21.15.88:0/974314526 --> v1:172.21.15.88:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f576c070c10 con 0x7f576c0a9ac0 2024-03-29T10:45:23.102 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:23.119+0000 7f5768ff9640 1 -- 172.21.15.88:0/974314526 <== mon.0 v1:172.21.15.88:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 344506374) 0x7f575c0524d0 con 0x7f576c0a9ac0 2024-03-29T10:45:23.102 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:23.119+0000 7f5768ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:45:23.102 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:23.119+0000 7f5768ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:45:23.102 INFO:teuthology.misc.health.smithi088.stdout: 2024-03-29T10:45:23.103 INFO:teuthology.misc.health.smithi088.stdout:{"epoch":10,"fsid":"869f5188-f92b-42dd-a46d-8c646e0388a9","created":"2024-03-29T10:40:40.285087+0000","modified":"2024-03-29T10:40:42.536230+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:40:42.536230+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a26c0c1c-8de0-438c-a3f6-3df10ea61574","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5de24915-8dec-49b6-8b16-f3170197c591","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ac054af3-d589-4565-9644-e6af75975c53","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3ec4f045-c714-4a22-bd1a-e07330b45ab3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"cddb9867-8e30-4951-9b13-3d1e78b9c45a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dc6d9911-785d-432f-8b53-56a1ccab302f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"77c88437-b587-496e-8b4d-1824ea8b5305","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ce963d96-5a8e-44c3-99de-c0dbebe234ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:45:23.108 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:23.123+0000 7f57527fc640 10 librados: watch_flush enter 2024-03-29T10:45:23.108 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:23.126+0000 7f57527fc640 10 librados: watch_flush exit 2024-03-29T10:45:23.108 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:23.126+0000 7f57527fc640 20 client.4389.objecter shutdown clearing up homeless session... 2024-03-29T10:45:23.108 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:23.126+0000 7f57527fc640 10 client.4389.objecter successfully canceled tick 2024-03-29T10:45:23.109 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:23.126+0000 7f57527fc640 1 -- 172.21.15.88:0/974314526 >> v1:172.21.15.88:6816/1752861349 conn(0x7f575806dc10 legacy=0x7f57580700d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:23.109 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:23.126+0000 7f57527fc640 10 monclient: shutdown 2024-03-29T10:45:23.109 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:23.126+0000 7f57527fc640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:45:23.109 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:23.126+0000 7f57527fc640 1 -- 172.21.15.88:0/974314526 >> v1:172.21.15.88:6789/0 conn(0x7f576c0a9ac0 legacy=0x7f576c070370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:23.109 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:23.126+0000 7f57527fc640 1 -- 172.21.15.88:0/974314526 shutdown_connections 2024-03-29T10:45:23.110 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:23.126+0000 7f57527fc640 1 -- 172.21.15.88:0/974314526 >> 172.21.15.88:0/974314526 conn(0x7f576c0b35e0 msgr2=0x7f576c0b07c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:45:23.112 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:23.129+0000 7f57527fc640 1 -- 172.21.15.88:0/974314526 shutdown_connections 2024-03-29T10:45:23.112 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:23.129+0000 7f57527fc640 1 -- 172.21.15.88:0/974314526 wait complete. 2024-03-29T10:45:23.112 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:23.129+0000 7f57527fc640 1 librados: shutdown 2024-03-29T10:45:23.131 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:45:29.937 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:45:30.086 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:30.104+0000 7f031b074640 10 monclient: get_monmap_and_config 2024-03-29T10:45:30.087 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:30.104+0000 7f031b074640 10 monclient: build_initial_monmap 2024-03-29T10:45:30.087 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:30.104+0000 7f031b074640 10 monclient: monmap: 2024-03-29T10:45:30.087 INFO:teuthology.misc.health.smithi088.stderr:epoch 0 2024-03-29T10:45:30.087 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:45:30.087 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:45:30.105087+0000 2024-03-29T10:45:30.087 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:45:30.105087+0000 2024-03-29T10:45:30.087 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 0 (unknown) 2024-03-29T10:45:30.087 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:45:30.087 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-03-29T10:45:30.087 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-29T10:45:30.087 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-b 2024-03-29T10:45:30.087 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:45:30.091 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:30.105+0000 7f031b074640 1 Processor -- start 2024-03-29T10:45:30.091 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:30.105+0000 7f031b074640 1 -- start start 2024-03-29T10:45:30.091 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:30.105+0000 7f031b074640 10 monclient: init 2024-03-29T10:45:30.091 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:30.106+0000 7f031b074640 10 monclient: _reopen_session rank -1 2024-03-29T10:45:30.092 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:30.106+0000 7f031b074640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:45:30.092 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:30.106+0000 7f031b074640 10 monclient(hunting): picked mon.noname-c con 0x7f03141457f0 addr v1:172.21.15.196:6789/0 2024-03-29T10:45:30.092 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:30.106+0000 7f031b074640 10 monclient(hunting): picked mon.noname-a con 0x7f031413f650 addr v1:172.21.15.88:6789/0 2024-03-29T10:45:30.092 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:30.106+0000 7f031b074640 10 monclient(hunting): picked mon.noname-b con 0x7f03140a9d70 addr v1:172.21.15.88:6790/0 2024-03-29T10:45:30.092 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:30.106+0000 7f031b074640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03140aaef0 con 0x7f031413f650 2024-03-29T10:45:30.092 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:30.106+0000 7f031b074640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0314145bd0 con 0x7f03141457f0 2024-03-29T10:45:30.092 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:30.106+0000 7f031b074640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0314145eb0 con 0x7f03140a9d70 2024-03-29T10:45:30.092 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:30.106+0000 7f031b074640 10 monclient(hunting): _renew_subs 2024-03-29T10:45:30.093 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:30.106+0000 7f031b074640 10 monclient(hunting): authenticate will time out at 1052.342480s 2024-03-29T10:45:30.093 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:30.110+0000 7f0318e10640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f03141457f0 0x7f031413ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.88:59927/0 (socket says 172.21.15.88:59927) 2024-03-29T10:45:30.093 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:30.110+0000 7f0318e10640 1 -- 172.21.15.88:0/2717065475 learned_addr learned my addr 172.21.15.88:0/2717065475 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-29T10:45:30.097 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:30.114+0000 7f03137fe640 1 -- 172.21.15.88:0/2717065475 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 878842964 0 0) 0x7f03140aaef0 con 0x7f031413f650 2024-03-29T10:45:30.097 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:30.114+0000 7f03137fe640 10 monclient(hunting): _negotiate 2024-03-29T10:45:30.097 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:30.114+0000 7f03137fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:45:30.097 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:30.114+0000 7f03137fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:45:30.097 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:30.114+0000 7f03137fe640 10 monclient(hunting): my global_id is 4392 2024-03-29T10:45:30.098 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:30.114+0000 7f03137fe640 1 -- 172.21.15.88:0/2717065475 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f02fc003520 con 0x7f031413f650 2024-03-29T10:45:30.098 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:30.114+0000 7f03137fe640 1 -- 172.21.15.88:0/2717065475 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2810918344 0 0) 0x7f0314145bd0 con 0x7f03141457f0 2024-03-29T10:45:30.098 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:30.114+0000 7f03137fe640 10 monclient(hunting): _negotiate 2024-03-29T10:45:30.099 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:30.114+0000 7f03137fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:45:30.099 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:30.114+0000 7f03137fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:45:30.099 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:30.114+0000 7f03137fe640 10 monclient(hunting): my global_id is 4393 2024-03-29T10:45:30.099 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:30.114+0000 7f03137fe640 1 -- 172.21.15.88:0/2717065475 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f03140aaef0 con 0x7f03141457f0 2024-03-29T10:45:30.100 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:30.114+0000 7f03137fe640 1 -- 172.21.15.88:0/2717065475 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3318805617 0 0) 0x7f02fc003520 con 0x7f031413f650 2024-03-29T10:45:30.100 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:30.114+0000 7f03137fe640 1 -- 172.21.15.88:0/2717065475 >> v1:172.21.15.88:6790/0 conn(0x7f03140a9d70 legacy=0x7f031414d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:30.101 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:30.114+0000 7f03137fe640 1 -- 172.21.15.88:0/2717065475 >> v1:172.21.15.196:6789/0 conn(0x7f03141457f0 legacy=0x7f031413ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:30.101 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:30.114+0000 7f03137fe640 10 monclient: _finish_hunting 0 2024-03-29T10:45:30.101 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:30.114+0000 7f03137fe640 1 monclient: found mon.noname-a 2024-03-29T10:45:30.101 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:30.114+0000 7f03137fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:45:30.101 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:30.114+0000 7f03137fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.88:6789/0 2024-03-29T10:45:30.102 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:30.114+0000 7f03137fe640 1 -- 172.21.15.88:0/2717065475 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0314146340 con 0x7f031413f650 2024-03-29T10:45:30.102 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:30.114+0000 7f03137fe640 10 monclient: _finish_auth 0 2024-03-29T10:45:30.102 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:30.115+0000 7f03137fe640 10 monclient: _check_auth_tickets 2024-03-29T10:45:30.102 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:30.115+0000 7f03137fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:45:30.103 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:30.115+0000 7f03137fe640 1 -- 172.21.15.88:0/2717065475 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f0304001690 con 0x7f031413f650 2024-03-29T10:45:30.103 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:30.115+0000 7f03137fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:30.103 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:30.115+0000 7f03137fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:45:30.103 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:30.115+0000 7f03137fe640 10 monclient: dump: 2024-03-29T10:45:30.104 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:45:30.104 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:45:30.104 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:45:30.104 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:45:30.105 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:30.105 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:45:30.105 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:45:30.105 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:45:30.106 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:45:30.106 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:45:30.106 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:30.115+0000 7f03137fe640 1 -- 172.21.15.88:0/2717065475 <== mon.0 v1:172.21.15.88:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f03040018b0 con 0x7f031413f650 2024-03-29T10:45:30.106 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:30.115+0000 7f03137fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:45:30.106 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:30.115+0000 7f03137fe640 1 -- 172.21.15.88:0/2717065475 <== mon.0 v1:172.21.15.88:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f0304001c40 con 0x7f031413f650 2024-03-29T10:45:30.106 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:30.115+0000 7f03137fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:30.107 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:30.115+0000 7f03137fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:45:30.107 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:30.115+0000 7f03137fe640 10 monclient: dump: 2024-03-29T10:45:30.107 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:45:30.107 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:45:30.107 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:45:30.107 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:45:30.107 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:30.107 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:45:30.107 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:45:30.107 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:45:30.107 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:45:30.107 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:45:30.107 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:30.117+0000 7f031b074640 5 monclient: authenticate success, global_id 4392 2024-03-29T10:45:30.107 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:30.117+0000 7f031b074640 10 monclient: get_monmap_and_config success 2024-03-29T10:45:30.107 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:30.117+0000 7f031b074640 4 set_mon_vals no callback set 2024-03-29T10:45:30.114 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:30.118+0000 7f031b074640 10 monclient: shutdown 2024-03-29T10:45:30.114 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:30.118+0000 7f031b074640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:45:30.114 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:30.118+0000 7f031b074640 1 -- 172.21.15.88:0/2717065475 >> v1:172.21.15.88:6789/0 conn(0x7f031413f650 legacy=0x7f03140b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:30.114 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:30.119+0000 7f031b074640 1 -- 172.21.15.88:0/2717065475 shutdown_connections 2024-03-29T10:45:30.115 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:30.119+0000 7f031b074640 1 -- 172.21.15.88:0/2717065475 >> 172.21.15.88:0/2717065475 conn(0x7f0314066a00 msgr2=0x7f0314066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:45:30.115 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:30.119+0000 7f031b074640 1 -- 172.21.15.88:0/2717065475 shutdown_connections 2024-03-29T10:45:30.115 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:30.119+0000 7f031b074640 1 -- 172.21.15.88:0/2717065475 wait complete. 2024-03-29T10:45:30.115 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:30.120+0000 7f031b074640 10 monclient: build_initial_monmap 2024-03-29T10:45:30.115 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:30.120+0000 7f031b074640 10 monclient: monmap: 2024-03-29T10:45:30.115 INFO:teuthology.misc.health.smithi088.stderr:epoch 0 2024-03-29T10:45:30.115 INFO:teuthology.misc.health.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:45:30.115 INFO:teuthology.misc.health.smithi088.stderr:last_changed 0.000000 2024-03-29T10:45:30.115 INFO:teuthology.misc.health.smithi088.stderr:created 0.000000 2024-03-29T10:45:30.115 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 0 (unknown) 2024-03-29T10:45:30.115 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:45:30.115 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-03-29T10:45:30.116 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-29T10:45:30.116 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-c 2024-03-29T10:45:30.116 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:45:30.116 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:30.120+0000 7f031b074640 1 librados: starting msgr at 2024-03-29T10:45:30.116 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:30.120+0000 7f031b074640 1 librados: starting objecter 2024-03-29T10:45:30.116 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:30.120+0000 7f031b074640 1 Processor -- start 2024-03-29T10:45:30.116 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:30.121+0000 7f031b074640 1 -- start start 2024-03-29T10:45:30.116 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:30.121+0000 7f031b074640 1 librados: setting wanted keys 2024-03-29T10:45:30.116 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:30.121+0000 7f031b074640 1 librados: calling monclient init 2024-03-29T10:45:30.116 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:30.121+0000 7f031b074640 10 monclient: init 2024-03-29T10:45:30.116 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:30.122+0000 7f031b074640 10 monclient: _reopen_session rank -1 2024-03-29T10:45:30.116 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:30.122+0000 7f031b074640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:45:30.116 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:30.122+0000 7f031b074640 10 monclient(hunting): picked mon.noname-b con 0x7f03140a9d70 addr v1:172.21.15.196:6789/0 2024-03-29T10:45:30.116 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:30.122+0000 7f031b074640 10 monclient(hunting): picked mon.noname-c con 0x7f031413f650 addr v1:172.21.15.88:6790/0 2024-03-29T10:45:30.117 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:30.122+0000 7f031b074640 10 monclient(hunting): picked mon.noname-a con 0x7f03141457f0 addr v1:172.21.15.88:6789/0 2024-03-29T10:45:30.117 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:30.122+0000 7f031b074640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0314141820 con 0x7f03141457f0 2024-03-29T10:45:30.117 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:30.122+0000 7f031b074640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f031406f800 con 0x7f03140a9d70 2024-03-29T10:45:30.117 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:30.122+0000 7f031b074640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f031406fae0 con 0x7f031413f650 2024-03-29T10:45:30.117 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:30.122+0000 7f031b074640 10 monclient(hunting): _renew_subs 2024-03-29T10:45:30.117 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:30.122+0000 7f031b074640 10 monclient(hunting): authenticate will time out at 1052.358140s 2024-03-29T10:45:30.117 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:30.125+0000 7f0319611640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f03141457f0 0x7f03141370f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:48995/0 (socket says 172.21.15.88:48995) 2024-03-29T10:45:30.117 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:30.126+0000 7f0319611640 1 -- 172.21.15.88:0/986677558 learned_addr learned my addr 172.21.15.88:0/986677558 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-29T10:45:30.117 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:30.126+0000 7f03117fa640 10 client.?.objecter ms_handle_connect 0x7f03141457f0 2024-03-29T10:45:30.117 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:30.126+0000 7f03117fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:45:30.117 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:30.126+0000 7f03117fa640 1 -- 172.21.15.88:0/986677558 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1810342213 0 0) 0x7f0314141820 con 0x7f03141457f0 2024-03-29T10:45:30.118 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:30.126+0000 7f03117fa640 10 monclient(hunting): _negotiate 2024-03-29T10:45:30.118 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:30.126+0000 7f03117fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:45:30.118 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:30.126+0000 7f03117fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:45:30.118 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:30.126+0000 7f03117fa640 10 monclient(hunting): my global_id is 4395 2024-03-29T10:45:30.118 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:30.126+0000 7f03117fa640 1 -- 172.21.15.88:0/986677558 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0300004eb0 con 0x7f03141457f0 2024-03-29T10:45:30.118 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:30.127+0000 7f03117fa640 1 -- 172.21.15.88:0/986677558 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3608538584 0 0) 0x7f0300004eb0 con 0x7f03141457f0 2024-03-29T10:45:30.118 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:30.127+0000 7f03117fa640 1 -- 172.21.15.88:0/986677558 --> v1:172.21.15.88:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0300005300 con 0x7f03141457f0 2024-03-29T10:45:30.118 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:30.127+0000 7f03117fa640 10 client.?.objecter ms_handle_connect 0x7f031413f650 2024-03-29T10:45:30.118 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:30.127+0000 7f03117fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:45:30.118 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:30.127+0000 7f03117fa640 1 -- 172.21.15.88:0/986677558 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f03080018a0 con 0x7f03141457f0 2024-03-29T10:45:30.118 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:30.127+0000 7f03117fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:30.119 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:30.127+0000 7f03117fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:45:30.119 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:30.127+0000 7f03117fa640 10 monclient(hunting): dump: 2024-03-29T10:45:30.119 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:45:30.119 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:45:30.119 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:45:30.119 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:45:30.119 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:30.119 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:45:30.119 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:45:30.119 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:45:30.119 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:45:30.119 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:45:30.120 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:30.127+0000 7f03117fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:45:30.120 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:30.127+0000 7f03117fa640 1 -- 172.21.15.88:0/986677558 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1189098061 0 0) 0x7f031406fae0 con 0x7f031413f650 2024-03-29T10:45:30.120 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:30.127+0000 7f03117fa640 10 monclient(hunting): _negotiate 2024-03-29T10:45:30.120 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:30.127+0000 7f03117fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:45:30.120 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:30.127+0000 7f03117fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:45:30.120 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:30.127+0000 7f03117fa640 10 monclient(hunting): my global_id is 4424 2024-03-29T10:45:30.120 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:30.127+0000 7f03117fa640 1 -- 172.21.15.88:0/986677558 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f03000058a0 con 0x7f031413f650 2024-03-29T10:45:30.120 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:30.128+0000 7f03117fa640 1 -- 172.21.15.88:0/986677558 <== mon.0 v1:172.21.15.88:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3503393080 0 0) 0x7f0300005300 con 0x7f03141457f0 2024-03-29T10:45:30.120 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:30.128+0000 7f03117fa640 1 -- 172.21.15.88:0/986677558 >> v1:172.21.15.88:6790/0 conn(0x7f031413f650 legacy=0x7f0314141110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:30.120 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:30.128+0000 7f03117fa640 1 -- 172.21.15.88:0/986677558 >> v1:172.21.15.196:6789/0 conn(0x7f03140a9d70 legacy=0x7f031406e100 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:45:30.120 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:30.129+0000 7f03117fa640 10 monclient: _finish_hunting 0 2024-03-29T10:45:30.120 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:30.129+0000 7f03117fa640 1 monclient: found mon.a 2024-03-29T10:45:30.120 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:30.129+0000 7f03117fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:45:30.120 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:30.129+0000 7f03117fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-03-29T10:45:30.121 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:30.129+0000 7f03117fa640 1 -- 172.21.15.88:0/986677558 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f031406fdc0 con 0x7f03141457f0 2024-03-29T10:45:30.121 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:30.129+0000 7f03117fa640 10 monclient: _finish_auth 0 2024-03-29T10:45:30.121 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:30.129+0000 7f03117fa640 10 monclient: _check_auth_tickets 2024-03-29T10:45:30.121 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:30.129+0000 7f03117fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:45:30.121 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:30.129+0000 7f031b074640 5 monclient: authenticate success, global_id 4395 2024-03-29T10:45:30.121 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:30.129+0000 7f031b074640 10 monclient: _renew_subs 2024-03-29T10:45:30.121 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:30.129+0000 7f031b074640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-03-29T10:45:30.121 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:30.129+0000 7f031b074640 1 -- 172.21.15.88:0/986677558 --> v1:172.21.15.88:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0314142d90 con 0x7f03141457f0 2024-03-29T10:45:30.121 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:30.129+0000 7f031b074640 10 client.4395.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:45:30.121 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:30.129+0000 7f031b074640 10 monclient: _renew_subs 2024-03-29T10:45:30.121 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:30.129+0000 7f031b074640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-03-29T10:45:30.121 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:30.129+0000 7f031b074640 1 -- 172.21.15.88:0/986677558 --> v1:172.21.15.88:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f03140701e0 con 0x7f03141457f0 2024-03-29T10:45:30.121 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:30.129+0000 7f031b074640 1 librados: init done 2024-03-29T10:45:30.122 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:30.129+0000 7f03117fa640 1 -- 172.21.15.88:0/986677558 <== mon.0 v1:172.21.15.88:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f03080035f0 con 0x7f03141457f0 2024-03-29T10:45:30.122 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:30.129+0000 7f03117fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:45:30.122 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:30.129+0000 7f0312ffd640 4 set_mon_vals no callback set 2024-03-29T10:45:30.122 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:30.130+0000 7f03117fa640 1 -- 172.21.15.88:0/986677558 <== mon.0 v1:172.21.15.88:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f03080018a0 con 0x7f03141457f0 2024-03-29T10:45:30.122 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:30.130+0000 7f03117fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:30.122 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:30.130+0000 7f03117fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:45:30.122 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:30.130+0000 7f03117fa640 10 monclient: dump: 2024-03-29T10:45:30.122 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:45:30.122 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:45:30.122 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:45:30.122 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:45:30.122 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:30.122 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:45:30.122 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:45:30.123 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:45:30.123 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:45:30.123 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:45:30.123 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:30.130+0000 7f031b074640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:45:30.123 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:30.130+0000 7f031b074640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:45:30.123 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:30.130+0000 7f031b074640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-03-29T10:45:30.123 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:30.130+0000 7f031b074640 1 -- 172.21.15.88:0/986677558 --> v1:172.21.15.88:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02f4005080 con 0x7f03141457f0 2024-03-29T10:45:30.124 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:30.135+0000 7f03117fa640 1 -- 172.21.15.88:0/986677558 <== mon.0 v1:172.21.15.88:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4235629220 0 0) 0x7f0308003760 con 0x7f03141457f0 2024-03-29T10:45:30.124 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:30.136+0000 7f0318e10640 10 monclient: get_auth_request con 0x7f030006e3f0 auth_method 0 2024-03-29T10:45:30.124 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:30.136+0000 7f03117fa640 10 client.4395.objecter ms_handle_connect 0x7f030006e3f0 2024-03-29T10:45:30.124 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:30.136+0000 7f03117fa640 1 -- 172.21.15.88:0/986677558 <== mon.0 v1:172.21.15.88:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4257673582 0 0) 0x7f03080543c0 con 0x7f03141457f0 2024-03-29T10:45:30.124 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:30.136+0000 7f03117fa640 10 client.4395.objecter ms_dispatch 0x7f0314000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:45:30.125 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:30.136+0000 7f03117fa640 3 client.4395.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:45:30.125 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:30.136+0000 7f03117fa640 3 client.4395.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:45:30.125 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:30.137+0000 7f03117fa640 20 client.4395.objecter dump_active .. 0 homeless 2024-03-29T10:45:30.126 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:30.144+0000 7f03117fa640 1 -- 172.21.15.88:0/986677558 <== mon.0 v1:172.21.15.88:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0308050660 con 0x7f03141457f0 2024-03-29T10:45:30.127 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:30.144+0000 7f03117fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:45:30.127 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:30.144+0000 7f03117fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:45:30.448 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:30.464+0000 7f031b074640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:45:30.448 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:30.465+0000 7f031b074640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:45:30.448 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:30.465+0000 7f031b074640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-03-29T10:45:30.448 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:30.465+0000 7f031b074640 1 -- 172.21.15.88:0/986677558 --> v1:172.21.15.88:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f02f4005330 con 0x7f03141457f0 2024-03-29T10:45:30.450 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:30.467+0000 7f03117fa640 1 -- 172.21.15.88:0/986677558 <== mon.0 v1:172.21.15.88:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 344506374) 0x7f0308050800 con 0x7f03141457f0 2024-03-29T10:45:30.450 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:30.467+0000 7f03117fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:45:30.450 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:30.467+0000 7f03117fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:45:30.450 INFO:teuthology.misc.health.smithi088.stdout: 2024-03-29T10:45:30.450 INFO:teuthology.misc.health.smithi088.stdout:{"epoch":10,"fsid":"869f5188-f92b-42dd-a46d-8c646e0388a9","created":"2024-03-29T10:40:40.285087+0000","modified":"2024-03-29T10:40:42.536230+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:40:42.536230+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a26c0c1c-8de0-438c-a3f6-3df10ea61574","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5de24915-8dec-49b6-8b16-f3170197c591","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ac054af3-d589-4565-9644-e6af75975c53","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3ec4f045-c714-4a22-bd1a-e07330b45ab3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"cddb9867-8e30-4951-9b13-3d1e78b9c45a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dc6d9911-785d-432f-8b53-56a1ccab302f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"77c88437-b587-496e-8b4d-1824ea8b5305","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ce963d96-5a8e-44c3-99de-c0dbebe234ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:45:30.453 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:30.470+0000 7f031b074640 10 librados: watch_flush enter 2024-03-29T10:45:30.454 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:30.471+0000 7f031b074640 10 librados: watch_flush exit 2024-03-29T10:45:30.454 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:30.471+0000 7f031b074640 20 client.4395.objecter shutdown clearing up homeless session... 2024-03-29T10:45:30.454 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:30.471+0000 7f031b074640 10 client.4395.objecter successfully canceled tick 2024-03-29T10:45:30.454 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:30.471+0000 7f031b074640 1 -- 172.21.15.88:0/986677558 >> v1:172.21.15.88:6816/1752861349 conn(0x7f030006e3f0 legacy=0x7f03000708b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:30.454 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:30.471+0000 7f031b074640 10 monclient: shutdown 2024-03-29T10:45:30.454 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:30.471+0000 7f031b074640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:45:30.454 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:30.471+0000 7f031b074640 1 -- 172.21.15.88:0/986677558 >> v1:172.21.15.88:6789/0 conn(0x7f03141457f0 legacy=0x7f03141370f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:30.455 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:30.471+0000 7f031b074640 1 -- 172.21.15.88:0/986677558 shutdown_connections 2024-03-29T10:45:30.455 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:30.471+0000 7f031b074640 1 -- 172.21.15.88:0/986677558 >> 172.21.15.88:0/986677558 conn(0x7f0314066a00 msgr2=0x7f0314066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:45:30.455 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:30.472+0000 7f031b074640 1 -- 172.21.15.88:0/986677558 shutdown_connections 2024-03-29T10:45:30.455 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:30.472+0000 7f031b074640 1 -- 172.21.15.88:0/986677558 wait complete. 2024-03-29T10:45:30.455 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:30.472+0000 7f031b074640 1 librados: shutdown 2024-03-29T10:45:30.472 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:45:37.278 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:45:37.426 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:37.443+0000 7f5027ba3640 10 monclient: get_monmap_and_config 2024-03-29T10:45:37.427 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:37.443+0000 7f5027ba3640 10 monclient: build_initial_monmap 2024-03-29T10:45:37.427 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:37.444+0000 7f5027ba3640 10 monclient: monmap: 2024-03-29T10:45:37.427 INFO:teuthology.misc.health.smithi088.stderr:epoch 0 2024-03-29T10:45:37.427 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:45:37.427 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:45:37.445028+0000 2024-03-29T10:45:37.427 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:45:37.445028+0000 2024-03-29T10:45:37.427 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 0 (unknown) 2024-03-29T10:45:37.427 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:45:37.427 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-03-29T10:45:37.427 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-29T10:45:37.427 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-b 2024-03-29T10:45:37.427 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:45:37.429 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:37.445+0000 7f5027ba3640 1 Processor -- start 2024-03-29T10:45:37.429 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:37.446+0000 7f5027ba3640 1 -- start start 2024-03-29T10:45:37.429 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:37.446+0000 7f5027ba3640 10 monclient: init 2024-03-29T10:45:37.433 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:37.447+0000 7f5027ba3640 10 monclient: _reopen_session rank -1 2024-03-29T10:45:37.433 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:37.447+0000 7f5027ba3640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:45:37.433 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:37.447+0000 7f5027ba3640 10 monclient(hunting): picked mon.noname-b con 0x7f50200b0860 addr v1:172.21.15.88:6790/0 2024-03-29T10:45:37.433 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:37.447+0000 7f5027ba3640 10 monclient(hunting): picked mon.noname-a con 0x7f502013ef30 addr v1:172.21.15.88:6789/0 2024-03-29T10:45:37.434 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:37.447+0000 7f5027ba3640 10 monclient(hunting): picked mon.noname-c con 0x7f50200af8e0 addr v1:172.21.15.196:6789/0 2024-03-29T10:45:37.434 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:37.447+0000 7f5027ba3640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50200b0c40 con 0x7f502013ef30 2024-03-29T10:45:37.434 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:37.447+0000 7f5027ba3640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50200aed10 con 0x7f50200af8e0 2024-03-29T10:45:37.434 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:37.447+0000 7f5027ba3640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50200afd50 con 0x7f50200b0860 2024-03-29T10:45:37.434 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:37.447+0000 7f5027ba3640 10 monclient(hunting): _renew_subs 2024-03-29T10:45:37.434 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:37.447+0000 7f5027ba3640 10 monclient(hunting): authenticate will time out at 1059.683084s 2024-03-29T10:45:37.434 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:37.451+0000 7f5026140640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f50200af8e0 0x7f5020149a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.88:51323/0 (socket says 172.21.15.88:51323) 2024-03-29T10:45:37.434 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:37.451+0000 7f5026140640 1 -- 172.21.15.88:0/1697521331 learned_addr learned my addr 172.21.15.88:0/1697521331 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-29T10:45:37.437 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:37.454+0000 7f502493d640 1 -- 172.21.15.88:0/1697521331 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2358175797 0 0) 0x7f50200aed10 con 0x7f50200af8e0 2024-03-29T10:45:37.437 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:37.454+0000 7f502493d640 10 monclient(hunting): _negotiate 2024-03-29T10:45:37.437 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:37.454+0000 7f502493d640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:45:37.437 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:37.454+0000 7f502493d640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:45:37.437 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:37.454+0000 7f502493d640 10 monclient(hunting): my global_id is 4396 2024-03-29T10:45:37.437 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:37.454+0000 7f502493d640 1 -- 172.21.15.88:0/1697521331 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f500c003520 con 0x7f50200af8e0 2024-03-29T10:45:37.437 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:37.454+0000 7f502493d640 1 -- 172.21.15.88:0/1697521331 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3229797046 0 0) 0x7f50200b0c40 con 0x7f502013ef30 2024-03-29T10:45:37.437 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:37.454+0000 7f502493d640 10 monclient(hunting): _negotiate 2024-03-29T10:45:37.437 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:37.454+0000 7f502493d640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:45:37.437 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:37.454+0000 7f502493d640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:45:37.437 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:37.454+0000 7f502493d640 10 monclient(hunting): my global_id is 4398 2024-03-29T10:45:37.438 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:37.454+0000 7f502493d640 1 -- 172.21.15.88:0/1697521331 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f50200aed10 con 0x7f502013ef30 2024-03-29T10:45:37.438 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:37.454+0000 7f502493d640 1 -- 172.21.15.88:0/1697521331 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3366974002 0 0) 0x7f50200afd50 con 0x7f50200b0860 2024-03-29T10:45:37.438 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:37.454+0000 7f502493d640 10 monclient(hunting): _negotiate 2024-03-29T10:45:37.438 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:37.454+0000 7f502493d640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:45:37.438 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:37.454+0000 7f502493d640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:45:37.438 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:37.454+0000 7f502493d640 10 monclient(hunting): my global_id is 4427 2024-03-29T10:45:37.438 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:37.454+0000 7f502493d640 1 -- 172.21.15.88:0/1697521331 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f50200b0c40 con 0x7f50200b0860 2024-03-29T10:45:37.438 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:37.454+0000 7f502493d640 1 -- 172.21.15.88:0/1697521331 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2349627936 0 0) 0x7f500c003520 con 0x7f50200af8e0 2024-03-29T10:45:37.438 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:37.454+0000 7f502493d640 1 -- 172.21.15.88:0/1697521331 >> v1:172.21.15.88:6790/0 conn(0x7f50200b0860 legacy=0x7f502013e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:37.438 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:37.454+0000 7f502493d640 1 -- 172.21.15.88:0/1697521331 >> v1:172.21.15.88:6789/0 conn(0x7f502013ef30 legacy=0x7f50200af1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:37.438 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:37.454+0000 7f502493d640 10 monclient: _finish_hunting 0 2024-03-29T10:45:37.438 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:37.454+0000 7f502493d640 1 monclient: found mon.noname-c 2024-03-29T10:45:37.438 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:37.454+0000 7f502493d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:45:37.438 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:37.454+0000 7f502493d640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.196:6789/0 2024-03-29T10:45:37.439 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:37.454+0000 7f502493d640 1 -- 172.21.15.88:0/1697521331 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50200ab1b0 con 0x7f50200af8e0 2024-03-29T10:45:37.439 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:37.454+0000 7f502493d640 10 monclient: _finish_auth 0 2024-03-29T10:45:37.439 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:37.454+0000 7f502493d640 10 monclient: _check_auth_tickets 2024-03-29T10:45:37.439 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:37.454+0000 7f502493d640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:45:37.439 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:37.454+0000 7f5027ba3640 5 monclient: authenticate success, global_id 4396 2024-03-29T10:45:37.439 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:37.454+0000 7f5027ba3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:45:37.439 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:37.454+0000 7f502493d640 1 -- 172.21.15.88:0/1697521331 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f5018005010 con 0x7f50200af8e0 2024-03-29T10:45:37.439 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:37.454+0000 7f502493d640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:37.439 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:37.454+0000 7f502493d640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:45:37.439 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:37.454+0000 7f502493d640 10 monclient: dump: 2024-03-29T10:45:37.439 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:45:37.439 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:45:37.439 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:45:37.439 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:45:37.440 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:37.440 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:45:37.440 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:45:37.440 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:45:37.440 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:45:37.440 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:45:37.440 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:37.454+0000 7f5027ba3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:45:37.441 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:37.457+0000 7f502493d640 1 -- 172.21.15.88:0/1697521331 <== mon.1 v1:172.21.15.196:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5018005170 con 0x7f50200af8e0 2024-03-29T10:45:37.441 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:37.457+0000 7f502493d640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:45:37.441 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:37.457+0000 7f5027ba3640 10 monclient: get_monmap_and_config success 2024-03-29T10:45:37.441 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:37.457+0000 7f5027ba3640 4 set_mon_vals no callback set 2024-03-29T10:45:37.441 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:37.457+0000 7f502493d640 1 -- 172.21.15.88:0/1697521331 <== mon.1 v1:172.21.15.196:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f5018001790 con 0x7f50200af8e0 2024-03-29T10:45:37.441 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:37.457+0000 7f502493d640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:37.441 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:37.457+0000 7f502493d640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:45:37.442 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:37.457+0000 7f502493d640 10 monclient: dump: 2024-03-29T10:45:37.442 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:45:37.442 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:45:37.442 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:45:37.442 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:45:37.442 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:37.442 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:45:37.442 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:45:37.442 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:45:37.442 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:45:37.442 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:45:37.442 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:37.458+0000 7f5027ba3640 10 monclient: shutdown 2024-03-29T10:45:37.442 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:37.458+0000 7f5027ba3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:45:37.442 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:37.458+0000 7f5027ba3640 1 -- 172.21.15.88:0/1697521331 >> v1:172.21.15.196:6789/0 conn(0x7f50200af8e0 legacy=0x7f5020149a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:37.443 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:37.458+0000 7f5027ba3640 1 -- 172.21.15.88:0/1697521331 shutdown_connections 2024-03-29T10:45:37.443 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:37.458+0000 7f5027ba3640 1 -- 172.21.15.88:0/1697521331 >> 172.21.15.88:0/1697521331 conn(0x7f502006b660 msgr2=0x7f502006ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:45:37.443 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:37.458+0000 7f5027ba3640 1 -- 172.21.15.88:0/1697521331 shutdown_connections 2024-03-29T10:45:37.443 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:37.458+0000 7f5027ba3640 1 -- 172.21.15.88:0/1697521331 wait complete. 2024-03-29T10:45:37.444 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:37.459+0000 7f5027ba3640 10 monclient: build_initial_monmap 2024-03-29T10:45:37.444 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:37.459+0000 7f5027ba3640 10 monclient: monmap: 2024-03-29T10:45:37.444 INFO:teuthology.misc.health.smithi088.stderr:epoch 0 2024-03-29T10:45:37.444 INFO:teuthology.misc.health.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:45:37.444 INFO:teuthology.misc.health.smithi088.stderr:last_changed 0.000000 2024-03-29T10:45:37.444 INFO:teuthology.misc.health.smithi088.stderr:created 0.000000 2024-03-29T10:45:37.444 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 0 (unknown) 2024-03-29T10:45:37.444 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:45:37.444 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-03-29T10:45:37.444 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-29T10:45:37.444 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-c 2024-03-29T10:45:37.444 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:45:37.444 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:37.459+0000 7f5027ba3640 1 librados: starting msgr at 2024-03-29T10:45:37.444 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:37.459+0000 7f5027ba3640 1 librados: starting objecter 2024-03-29T10:45:37.445 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:37.460+0000 7f5027ba3640 1 Processor -- start 2024-03-29T10:45:37.447 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:37.460+0000 7f5027ba3640 1 -- start start 2024-03-29T10:45:37.447 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:37.460+0000 7f5027ba3640 1 librados: setting wanted keys 2024-03-29T10:45:37.447 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:37.460+0000 7f5027ba3640 1 librados: calling monclient init 2024-03-29T10:45:37.447 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:37.460+0000 7f5027ba3640 10 monclient: init 2024-03-29T10:45:37.447 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:37.460+0000 7f5027ba3640 10 monclient: _reopen_session rank -1 2024-03-29T10:45:37.447 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:37.460+0000 7f5027ba3640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:45:37.448 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:37.463+0000 7f5027ba3640 10 monclient(hunting): picked mon.noname-a con 0x7f50200af8e0 addr v1:172.21.15.88:6789/0 2024-03-29T10:45:37.448 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:37.463+0000 7f5027ba3640 10 monclient(hunting): picked mon.noname-b con 0x7f50200b0860 addr v1:172.21.15.196:6789/0 2024-03-29T10:45:37.448 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:37.463+0000 7f5027ba3640 10 monclient(hunting): picked mon.noname-c con 0x7f502013ef30 addr v1:172.21.15.88:6790/0 2024-03-29T10:45:37.448 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:37.463+0000 7f5027ba3640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50200b1b50 con 0x7f50200af8e0 2024-03-29T10:45:37.448 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:37.463+0000 7f5027ba3640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50200b2d40 con 0x7f50200b0860 2024-03-29T10:45:37.448 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:37.463+0000 7f5027ba3640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50200b4170 con 0x7f502013ef30 2024-03-29T10:45:37.448 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:37.463+0000 7f5027ba3640 10 monclient(hunting): _renew_subs 2024-03-29T10:45:37.448 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:37.463+0000 7f5027ba3640 10 monclient(hunting): authenticate will time out at 1059.699638s 2024-03-29T10:45:37.450 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:37.466+0000 7f502593f640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f50200af8e0 0x7f50200b1440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:43849/0 (socket says 172.21.15.88:43849) 2024-03-29T10:45:37.450 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:37.466+0000 7f502593f640 1 -- 172.21.15.88:0/2372236886 learned_addr learned my addr 172.21.15.88:0/2372236886 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-29T10:45:37.450 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:37.467+0000 7f50167fc640 10 client.?.objecter ms_handle_connect 0x7f50200af8e0 2024-03-29T10:45:37.450 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:37.467+0000 7f50167fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:45:37.453 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:37.468+0000 7f50167fc640 10 client.?.objecter ms_handle_connect 0x7f502013ef30 2024-03-29T10:45:37.453 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:37.468+0000 7f50167fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:45:37.453 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:37.468+0000 7f50167fc640 1 -- 172.21.15.88:0/2372236886 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3882405114 0 0) 0x7f50200b4170 con 0x7f502013ef30 2024-03-29T10:45:37.453 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:37.468+0000 7f50167fc640 10 monclient(hunting): _negotiate 2024-03-29T10:45:37.453 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:37.468+0000 7f50167fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:45:37.453 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:37.468+0000 7f50167fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:45:37.453 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:37.468+0000 7f50167fc640 10 monclient(hunting): my global_id is 4430 2024-03-29T10:45:37.453 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:37.468+0000 7f50167fc640 1 -- 172.21.15.88:0/2372236886 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5004004eb0 con 0x7f502013ef30 2024-03-29T10:45:37.453 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:37.468+0000 7f50167fc640 1 -- 172.21.15.88:0/2372236886 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3739842027 0 0) 0x7f5004004eb0 con 0x7f502013ef30 2024-03-29T10:45:37.453 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:37.468+0000 7f50167fc640 1 -- 172.21.15.88:0/2372236886 --> v1:172.21.15.88:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5004005300 con 0x7f502013ef30 2024-03-29T10:45:37.454 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:37.469+0000 7f50167fc640 1 -- 172.21.15.88:0/2372236886 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f5018001580 con 0x7f502013ef30 2024-03-29T10:45:37.454 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:37.469+0000 7f50167fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:37.454 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:37.469+0000 7f50167fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:45:37.454 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:37.469+0000 7f50167fc640 10 monclient(hunting): dump: 2024-03-29T10:45:37.454 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:45:37.454 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:45:37.454 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:45:37.454 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:45:37.454 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:37.454 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:45:37.454 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:45:37.454 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:45:37.454 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:45:37.454 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:45:37.455 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:37.469+0000 7f50167fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:45:37.455 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:37.469+0000 7f50167fc640 1 -- 172.21.15.88:0/2372236886 <== mon.2 v1:172.21.15.88:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 744388416 0 0) 0x7f5004005300 con 0x7f502013ef30 2024-03-29T10:45:37.455 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:37.469+0000 7f50167fc640 1 -- 172.21.15.88:0/2372236886 >> v1:172.21.15.196:6789/0 conn(0x7f50200b0860 legacy=0x7f50201d34d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:37.455 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:37.469+0000 7f50167fc640 1 -- 172.21.15.88:0/2372236886 >> v1:172.21.15.88:6789/0 conn(0x7f50200af8e0 legacy=0x7f50200b1440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:37.455 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:37.469+0000 7f50167fc640 10 monclient: _finish_hunting 0 2024-03-29T10:45:37.455 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:37.469+0000 7f50167fc640 1 monclient: found mon.c 2024-03-29T10:45:37.455 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:37.469+0000 7f50167fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:45:37.455 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:37.469+0000 7f50167fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-03-29T10:45:37.455 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:37.469+0000 7f50167fc640 1 -- 172.21.15.88:0/2372236886 --> v1:172.21.15.88:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50200b4300 con 0x7f502013ef30 2024-03-29T10:45:37.455 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:37.469+0000 7f50167fc640 10 monclient: _finish_auth 0 2024-03-29T10:45:37.455 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:37.469+0000 7f50167fc640 10 monclient: _check_auth_tickets 2024-03-29T10:45:37.455 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:37.469+0000 7f50167fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:45:37.455 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:37.469+0000 7f5027ba3640 5 monclient: authenticate success, global_id 4430 2024-03-29T10:45:37.455 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:37.469+0000 7f5027ba3640 10 monclient: _renew_subs 2024-03-29T10:45:37.455 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:37.469+0000 7f5027ba3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-03-29T10:45:37.456 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:37.469+0000 7f5027ba3640 1 -- 172.21.15.88:0/2372236886 --> v1:172.21.15.88:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50201d3be0 con 0x7f502013ef30 2024-03-29T10:45:37.456 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:37.469+0000 7f5027ba3640 10 client.4430.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:45:37.456 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:37.469+0000 7f5027ba3640 10 monclient: _renew_subs 2024-03-29T10:45:37.456 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:37.469+0000 7f5027ba3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-03-29T10:45:37.456 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:37.469+0000 7f5027ba3640 1 -- 172.21.15.88:0/2372236886 --> v1:172.21.15.88:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f502013bb50 con 0x7f502013ef30 2024-03-29T10:45:37.456 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:37.469+0000 7f5027ba3640 1 librados: init done 2024-03-29T10:45:37.457 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:37.472+0000 7f5027ba3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:45:37.457 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:37.472+0000 7f50167fc640 1 -- 172.21.15.88:0/2372236886 <== mon.2 v1:172.21.15.88:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5018001b30 con 0x7f502013ef30 2024-03-29T10:45:37.457 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:37.472+0000 7f50167fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:45:37.457 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:37.472+0000 7f5017fff640 4 set_mon_vals no callback set 2024-03-29T10:45:37.457 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:37.472+0000 7f50167fc640 1 -- 172.21.15.88:0/2372236886 <== mon.2 v1:172.21.15.88:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f5018001580 con 0x7f502013ef30 2024-03-29T10:45:37.457 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:37.474+0000 7f5027ba3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:45:37.457 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:37.474+0000 7f5027ba3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-03-29T10:45:37.458 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:37.474+0000 7f5027ba3640 1 -- 172.21.15.88:0/2372236886 --> v1:172.21.15.88:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5020075f10 con 0x7f502013ef30 2024-03-29T10:45:37.458 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:37.474+0000 7f50167fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:37.458 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:37.474+0000 7f50167fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:45:37.458 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:37.474+0000 7f50167fc640 10 monclient: dump: 2024-03-29T10:45:37.458 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:45:37.458 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:45:37.458 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:45:37.458 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:45:37.458 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:37.458 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:45:37.458 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:45:37.458 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:45:37.458 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:45:37.458 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:45:37.459 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:37.475+0000 7f50167fc640 1 -- 172.21.15.88:0/2372236886 <== mon.2 v1:172.21.15.88:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4235629220 0 0) 0x7f5018001de0 con 0x7f502013ef30 2024-03-29T10:45:37.460 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:37.476+0000 7f50167fc640 1 -- 172.21.15.88:0/2372236886 <== mon.2 v1:172.21.15.88:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4257673582 0 0) 0x7f5018084b30 con 0x7f502013ef30 2024-03-29T10:45:37.460 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:37.476+0000 7f50167fc640 10 client.4430.objecter ms_dispatch 0x7f5020000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:45:37.460 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:37.476+0000 7f50167fc640 3 client.4430.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:45:37.460 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:37.476+0000 7f50167fc640 3 client.4430.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:45:37.460 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:37.476+0000 7f50167fc640 20 client.4430.objecter dump_active .. 0 homeless 2024-03-29T10:45:37.460 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:37.476+0000 7f502593f640 10 monclient: get_auth_request con 0x7f5004072420 auth_method 0 2024-03-29T10:45:37.460 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:37.476+0000 7f50167fc640 10 client.4430.objecter ms_handle_connect 0x7f5004072420 2024-03-29T10:45:37.466 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:37.483+0000 7f50167fc640 1 -- 172.21.15.88:0/2372236886 <== mon.2 v1:172.21.15.88:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5018050470 con 0x7f502013ef30 2024-03-29T10:45:37.466 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:37.483+0000 7f50167fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:45:37.466 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:37.483+0000 7f50167fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:45:37.788 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:37.806+0000 7f5027ba3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:45:37.789 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:37.806+0000 7f5027ba3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:45:37.789 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:37.806+0000 7f5027ba3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-03-29T10:45:37.789 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:37.806+0000 7f5027ba3640 1 -- 172.21.15.88:0/2372236886 --> v1:172.21.15.88:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5020075f10 con 0x7f502013ef30 2024-03-29T10:45:37.792 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:37.810+0000 7f50167fc640 1 -- 172.21.15.88:0/2372236886 <== mon.2 v1:172.21.15.88:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 344506374) 0x7f50180523a0 con 0x7f502013ef30 2024-03-29T10:45:37.792 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:37.810+0000 7f50167fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:45:37.792 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:37.810+0000 7f50167fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:45:37.795 INFO:teuthology.misc.health.smithi088.stdout: 2024-03-29T10:45:37.795 INFO:teuthology.misc.health.smithi088.stdout:{"epoch":10,"fsid":"869f5188-f92b-42dd-a46d-8c646e0388a9","created":"2024-03-29T10:40:40.285087+0000","modified":"2024-03-29T10:40:42.536230+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:40:42.536230+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a26c0c1c-8de0-438c-a3f6-3df10ea61574","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5de24915-8dec-49b6-8b16-f3170197c591","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ac054af3-d589-4565-9644-e6af75975c53","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3ec4f045-c714-4a22-bd1a-e07330b45ab3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"cddb9867-8e30-4951-9b13-3d1e78b9c45a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dc6d9911-785d-432f-8b53-56a1ccab302f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"77c88437-b587-496e-8b4d-1824ea8b5305","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ce963d96-5a8e-44c3-99de-c0dbebe234ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:45:37.800 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:37.815+0000 7f5027ba3640 10 librados: watch_flush enter 2024-03-29T10:45:37.800 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:37.817+0000 7f5027ba3640 10 librados: watch_flush exit 2024-03-29T10:45:37.800 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:37.817+0000 7f5027ba3640 20 client.4430.objecter shutdown clearing up homeless session... 2024-03-29T10:45:37.800 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:37.817+0000 7f5027ba3640 10 client.4430.objecter successfully canceled tick 2024-03-29T10:45:37.800 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:37.817+0000 7f5027ba3640 1 -- 172.21.15.88:0/2372236886 >> v1:172.21.15.88:6816/1752861349 conn(0x7f5004072420 legacy=0x7f50040748e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:37.800 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:37.817+0000 7f5027ba3640 10 monclient: shutdown 2024-03-29T10:45:37.800 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:37.817+0000 7f5027ba3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:45:37.800 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:37.817+0000 7f5027ba3640 1 -- 172.21.15.88:0/2372236886 >> v1:172.21.15.88:6790/0 conn(0x7f502013ef30 legacy=0x7f50201dd800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:37.801 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:37.817+0000 7f5027ba3640 1 -- 172.21.15.88:0/2372236886 shutdown_connections 2024-03-29T10:45:37.801 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:37.817+0000 7f5027ba3640 1 -- 172.21.15.88:0/2372236886 >> 172.21.15.88:0/2372236886 conn(0x7f502006b660 msgr2=0x7f502013ddb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:45:37.802 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:37.819+0000 7f5027ba3640 1 -- 172.21.15.88:0/2372236886 shutdown_connections 2024-03-29T10:45:37.804 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:37.821+0000 7f5027ba3640 1 -- 172.21.15.88:0/2372236886 wait complete. 2024-03-29T10:45:37.804 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:37.821+0000 7f5027ba3640 1 librados: shutdown 2024-03-29T10:45:37.825 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:45:44.632 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:45:44.784 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:44.801+0000 7f3240c15640 10 monclient: get_monmap_and_config 2024-03-29T10:45:44.784 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:44.801+0000 7f3240c15640 10 monclient: build_initial_monmap 2024-03-29T10:45:44.784 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:44.801+0000 7f3240c15640 10 monclient: monmap: 2024-03-29T10:45:44.784 INFO:teuthology.misc.health.smithi088.stderr:epoch 0 2024-03-29T10:45:44.784 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:45:44.785 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:45:44.802834+0000 2024-03-29T10:45:44.785 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:45:44.802834+0000 2024-03-29T10:45:44.785 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 0 (unknown) 2024-03-29T10:45:44.785 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:45:44.785 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-03-29T10:45:44.785 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-29T10:45:44.785 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-b 2024-03-29T10:45:44.785 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:45:44.785 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:44.802+0000 7f3240c15640 1 Processor -- start 2024-03-29T10:45:44.785 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:44.803+0000 7f3240c15640 1 -- start start 2024-03-29T10:45:44.786 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:44.803+0000 7f3240c15640 10 monclient: init 2024-03-29T10:45:44.787 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:44.804+0000 7f3240c15640 10 monclient: _reopen_session rank -1 2024-03-29T10:45:44.788 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:44.804+0000 7f3240c15640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:45:44.790 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:44.807+0000 7f3240c15640 10 monclient(hunting): picked mon.noname-c con 0x7f323c0b1f60 addr v1:172.21.15.196:6789/0 2024-03-29T10:45:44.790 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:44.807+0000 7f3240c15640 10 monclient(hunting): picked mon.noname-b con 0x7f323c071eb0 addr v1:172.21.15.88:6790/0 2024-03-29T10:45:44.790 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:44.807+0000 7f3240c15640 10 monclient(hunting): picked mon.noname-a con 0x7f323c143d10 addr v1:172.21.15.88:6789/0 2024-03-29T10:45:44.790 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:44.807+0000 7f3240c15640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f323c06ef80 con 0x7f323c143d10 2024-03-29T10:45:44.790 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:44.807+0000 7f3240c15640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f323c06f260 con 0x7f323c0b1f60 2024-03-29T10:45:44.790 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:44.807+0000 7f3240c15640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f323c0b40c0 con 0x7f323c071eb0 2024-03-29T10:45:44.790 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:44.807+0000 7f3240c15640 10 monclient(hunting): _renew_subs 2024-03-29T10:45:44.791 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:44.807+0000 7f3240c15640 10 monclient(hunting): authenticate will time out at 1067.043651s 2024-03-29T10:45:44.793 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:44.810+0000 7f323a59c640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f323c0b1f60 0x7f323c143600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.88:47135/0 (socket says 172.21.15.88:47135) 2024-03-29T10:45:44.793 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:44.810+0000 7f323a59c640 1 -- 172.21.15.88:0/3952189674 learned_addr learned my addr 172.21.15.88:0/3952189674 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-29T10:45:44.793 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:44.811+0000 7f323959a640 1 -- 172.21.15.88:0/3952189674 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3507775419 0 0) 0x7f323c06f260 con 0x7f323c0b1f60 2024-03-29T10:45:44.793 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:44.811+0000 7f323959a640 10 monclient(hunting): _negotiate 2024-03-29T10:45:44.794 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:44.811+0000 7f323959a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:45:44.794 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:44.811+0000 7f323959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:45:44.794 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:44.811+0000 7f323959a640 10 monclient(hunting): my global_id is 4399 2024-03-29T10:45:44.794 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:44.811+0000 7f323959a640 1 -- 172.21.15.88:0/3952189674 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3224003520 con 0x7f323c0b1f60 2024-03-29T10:45:44.794 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:44.811+0000 7f323959a640 1 -- 172.21.15.88:0/3952189674 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1012833163 0 0) 0x7f3224003520 con 0x7f323c0b1f60 2024-03-29T10:45:44.794 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:44.811+0000 7f323959a640 1 -- 172.21.15.88:0/3952189674 >> v1:172.21.15.88:6790/0 conn(0x7f323c071eb0 legacy=0x7f323c146470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:45:44.794 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:44.811+0000 7f323959a640 1 -- 172.21.15.88:0/3952189674 >> v1:172.21.15.88:6789/0 conn(0x7f323c143d10 legacy=0x7f323c146b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:44.794 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:44.811+0000 7f323959a640 10 monclient: _finish_hunting 0 2024-03-29T10:45:44.795 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:44.811+0000 7f323959a640 1 monclient: found mon.noname-c 2024-03-29T10:45:44.795 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:44.811+0000 7f323959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:45:44.795 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:44.811+0000 7f323959a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.196:6789/0 2024-03-29T10:45:44.795 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:44.811+0000 7f323959a640 1 -- 172.21.15.88:0/3952189674 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f323c155d70 con 0x7f323c0b1f60 2024-03-29T10:45:44.795 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:44.811+0000 7f323959a640 10 monclient: _finish_auth 0 2024-03-29T10:45:44.795 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:44.811+0000 7f323959a640 10 monclient: _check_auth_tickets 2024-03-29T10:45:44.795 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:44.811+0000 7f323959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:45:44.795 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:44.811+0000 7f323959a640 1 -- 172.21.15.88:0/3952189674 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f322c001690 con 0x7f323c0b1f60 2024-03-29T10:45:44.795 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:44.811+0000 7f323959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:44.795 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:44.811+0000 7f323959a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:45:44.795 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:44.811+0000 7f323959a640 10 monclient: dump: 2024-03-29T10:45:44.795 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:45:44.796 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:45:44.796 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:45:44.796 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:45:44.796 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:44.796 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:45:44.796 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:45:44.796 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:45:44.796 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:45:44.796 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:45:44.796 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:44.811+0000 7f3240c15640 5 monclient: authenticate success, global_id 4399 2024-03-29T10:45:44.796 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:44.811+0000 7f3240c15640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:45:44.797 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:44.813+0000 7f323959a640 1 -- 172.21.15.88:0/3952189674 <== mon.1 v1:172.21.15.196:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f322c001860 con 0x7f323c0b1f60 2024-03-29T10:45:44.797 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:44.813+0000 7f323959a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:45:44.797 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:44.813+0000 7f323959a640 1 -- 172.21.15.88:0/3952189674 <== mon.1 v1:172.21.15.196:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f322c001c70 con 0x7f323c0b1f60 2024-03-29T10:45:44.797 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:44.813+0000 7f323959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:44.797 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:44.813+0000 7f323959a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:45:44.797 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:44.813+0000 7f323959a640 10 monclient: dump: 2024-03-29T10:45:44.797 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:45:44.797 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:45:44.797 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:45:44.798 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:45:44.798 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:44.798 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:45:44.798 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:45:44.798 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:45:44.798 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:45:44.798 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:45:44.798 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:44.813+0000 7f3240c15640 10 monclient: get_monmap_and_config success 2024-03-29T10:45:44.798 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:44.813+0000 7f3240c15640 4 set_mon_vals no callback set 2024-03-29T10:45:44.799 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:44.815+0000 7f3240c15640 10 monclient: shutdown 2024-03-29T10:45:44.799 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:44.815+0000 7f3240c15640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:45:44.799 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:44.815+0000 7f3240c15640 1 -- 172.21.15.88:0/3952189674 >> v1:172.21.15.196:6789/0 conn(0x7f323c0b1f60 legacy=0x7f323c143600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:44.800 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:44.818+0000 7f3240c15640 1 -- 172.21.15.88:0/3952189674 shutdown_connections 2024-03-29T10:45:44.800 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:44.818+0000 7f3240c15640 1 -- 172.21.15.88:0/3952189674 >> 172.21.15.88:0/3952189674 conn(0x7f323c066a00 msgr2=0x7f323c066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:45:44.801 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:44.819+0000 7f3240c15640 1 -- 172.21.15.88:0/3952189674 shutdown_connections 2024-03-29T10:45:44.801 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:44.819+0000 7f3240c15640 1 -- 172.21.15.88:0/3952189674 wait complete. 2024-03-29T10:45:44.806 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:44.820+0000 7f3240c15640 10 monclient: build_initial_monmap 2024-03-29T10:45:44.806 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:44.820+0000 7f3240c15640 10 monclient: monmap: 2024-03-29T10:45:44.807 INFO:teuthology.misc.health.smithi088.stderr:epoch 0 2024-03-29T10:45:44.807 INFO:teuthology.misc.health.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:45:44.807 INFO:teuthology.misc.health.smithi088.stderr:last_changed 0.000000 2024-03-29T10:45:44.807 INFO:teuthology.misc.health.smithi088.stderr:created 0.000000 2024-03-29T10:45:44.807 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 0 (unknown) 2024-03-29T10:45:44.807 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:45:44.807 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-03-29T10:45:44.807 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-29T10:45:44.807 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-c 2024-03-29T10:45:44.807 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:45:44.807 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:44.820+0000 7f3240c15640 1 librados: starting msgr at 2024-03-29T10:45:44.807 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:44.820+0000 7f3240c15640 1 librados: starting objecter 2024-03-29T10:45:44.808 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:44.821+0000 7f3240c15640 1 Processor -- start 2024-03-29T10:45:44.808 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:44.821+0000 7f3240c15640 1 -- start start 2024-03-29T10:45:44.808 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:44.821+0000 7f3240c15640 1 librados: setting wanted keys 2024-03-29T10:45:44.808 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:44.821+0000 7f3240c15640 1 librados: calling monclient init 2024-03-29T10:45:44.808 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:44.821+0000 7f3240c15640 10 monclient: init 2024-03-29T10:45:44.808 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:44.821+0000 7f3240c15640 10 monclient: _reopen_session rank -1 2024-03-29T10:45:44.808 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:44.821+0000 7f3240c15640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:45:44.808 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:44.821+0000 7f3240c15640 10 monclient(hunting): picked mon.noname-b con 0x7f323c0b1f60 addr v1:172.21.15.196:6789/0 2024-03-29T10:45:44.808 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:44.824+0000 7f3240c15640 10 monclient(hunting): picked mon.noname-a con 0x7f323c143d10 addr v1:172.21.15.88:6789/0 2024-03-29T10:45:44.808 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:44.824+0000 7f3240c15640 10 monclient(hunting): picked mon.noname-c con 0x7f323c1547a0 addr v1:172.21.15.88:6790/0 2024-03-29T10:45:44.808 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:44.824+0000 7f3240c15640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f323c0aca60 con 0x7f323c143d10 2024-03-29T10:45:44.808 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:44.824+0000 7f3240c15640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f323c140820 con 0x7f323c0b1f60 2024-03-29T10:45:44.809 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:44.824+0000 7f3240c15640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f323c0ab460 con 0x7f323c1547a0 2024-03-29T10:45:44.809 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:44.824+0000 7f3240c15640 10 monclient(hunting): _renew_subs 2024-03-29T10:45:44.809 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:44.824+0000 7f3240c15640 10 monclient(hunting): authenticate will time out at 1067.060211s 2024-03-29T10:45:44.809 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:44.825+0000 7f323a59c640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f323c0b1f60 0x7f323c1e3d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.88:47245/0 (socket says 172.21.15.88:47245) 2024-03-29T10:45:44.809 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:44.825+0000 7f323a59c640 1 -- 172.21.15.88:0/1427160741 learned_addr learned my addr 172.21.15.88:0/1427160741 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-29T10:45:44.809 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:44.826+0000 7f322affd640 10 client.?.objecter ms_handle_connect 0x7f323c0b1f60 2024-03-29T10:45:44.809 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:44.826+0000 7f322affd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:45:44.809 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:44.826+0000 7f322affd640 1 -- 172.21.15.88:0/1427160741 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3727368616 0 0) 0x7f323c140820 con 0x7f323c0b1f60 2024-03-29T10:45:44.810 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:44.826+0000 7f322affd640 10 monclient(hunting): _negotiate 2024-03-29T10:45:44.810 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:44.826+0000 7f322affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:45:44.810 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:44.826+0000 7f322affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:45:44.810 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:44.826+0000 7f322affd640 10 monclient(hunting): my global_id is 4402 2024-03-29T10:45:44.810 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:44.826+0000 7f322affd640 1 -- 172.21.15.88:0/1427160741 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f321c004eb0 con 0x7f323c0b1f60 2024-03-29T10:45:44.811 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:44.828+0000 7f322affd640 10 client.?.objecter ms_handle_connect 0x7f323c143d10 2024-03-29T10:45:44.811 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:44.828+0000 7f322affd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:45:44.811 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:44.828+0000 7f322affd640 1 -- 172.21.15.88:0/1427160741 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2766886490 0 0) 0x7f321c004eb0 con 0x7f323c0b1f60 2024-03-29T10:45:44.811 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:44.828+0000 7f322affd640 1 -- 172.21.15.88:0/1427160741 --> v1:172.21.15.196:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f323c140820 con 0x7f323c0b1f60 2024-03-29T10:45:44.811 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:44.828+0000 7f322affd640 1 -- 172.21.15.88:0/1427160741 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f322c001b70 con 0x7f323c0b1f60 2024-03-29T10:45:44.811 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:44.828+0000 7f322affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:44.811 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:44.828+0000 7f322affd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:45:44.811 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:44.828+0000 7f322affd640 10 monclient(hunting): dump: 2024-03-29T10:45:44.811 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:45:44.811 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:45:44.811 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:45:44.811 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:45:44.812 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:44.812 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:45:44.812 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:45:44.812 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:45:44.812 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:45:44.812 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:45:44.812 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:44.828+0000 7f322affd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:45:44.812 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:44.828+0000 7f322affd640 1 -- 172.21.15.88:0/1427160741 <== mon.1 v1:172.21.15.196:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 28001381 0 0) 0x7f323c140820 con 0x7f323c0b1f60 2024-03-29T10:45:44.812 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:44.828+0000 7f322affd640 1 -- 172.21.15.88:0/1427160741 >> v1:172.21.15.88:6790/0 conn(0x7f323c1547a0 legacy=0x7f323c140110 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:45:44.814 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:44.831+0000 7f322affd640 1 -- 172.21.15.88:0/1427160741 >> v1:172.21.15.88:6789/0 conn(0x7f323c143d10 legacy=0x7f323c13fa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:44.814 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:44.831+0000 7f322affd640 10 monclient: _finish_hunting 0 2024-03-29T10:45:44.814 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:44.831+0000 7f322affd640 1 monclient: found mon.b 2024-03-29T10:45:44.814 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:44.831+0000 7f322affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:45:44.814 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:44.831+0000 7f322affd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T10:45:44.814 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:44.831+0000 7f322affd640 1 -- 172.21.15.88:0/1427160741 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f323c0accb0 con 0x7f323c0b1f60 2024-03-29T10:45:44.814 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:44.831+0000 7f322affd640 10 monclient: _finish_auth 0 2024-03-29T10:45:44.815 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:44.831+0000 7f322affd640 10 monclient: _check_auth_tickets 2024-03-29T10:45:44.815 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:44.831+0000 7f322affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:45:44.815 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:44.831+0000 7f322affd640 10 client.?.objecter ms_handle_connect 0x7f323c1547a0 2024-03-29T10:45:44.815 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:44.831+0000 7f322affd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:45:44.815 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:44.831+0000 7f3240c15640 5 monclient: authenticate success, global_id 4402 2024-03-29T10:45:44.815 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:44.831+0000 7f3240c15640 10 monclient: _renew_subs 2024-03-29T10:45:44.815 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:44.831+0000 7f3240c15640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T10:45:44.815 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:44.831+0000 7f3240c15640 1 -- 172.21.15.88:0/1427160741 --> v1:172.21.15.196:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f323c1528b0 con 0x7f323c0b1f60 2024-03-29T10:45:44.815 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:44.831+0000 7f3240c15640 10 client.4402.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:45:44.815 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:44.831+0000 7f3240c15640 10 monclient: _renew_subs 2024-03-29T10:45:44.815 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:44.831+0000 7f3240c15640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T10:45:44.815 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:44.831+0000 7f3240c15640 1 -- 172.21.15.88:0/1427160741 --> v1:172.21.15.196:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f323c0aa8c0 con 0x7f323c0b1f60 2024-03-29T10:45:44.815 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:44.831+0000 7f3240c15640 1 librados: init done 2024-03-29T10:45:44.816 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:44.833+0000 7f3240c15640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:45:44.816 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:44.833+0000 7f3240c15640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:45:44.816 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:44.833+0000 7f3240c15640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T10:45:44.816 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:44.833+0000 7f3240c15640 1 -- 172.21.15.88:0/1427160741 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f323c075f10 con 0x7f323c0b1f60 2024-03-29T10:45:44.816 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:44.833+0000 7f322affd640 1 -- 172.21.15.88:0/1427160741 <== mon.1 v1:172.21.15.196:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f322c004bf0 con 0x7f323c0b1f60 2024-03-29T10:45:44.818 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:44.833+0000 7f322affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:45:44.818 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:44.833+0000 7f322affd640 1 -- 172.21.15.88:0/1427160741 <== mon.1 v1:172.21.15.196:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f322c004f00 con 0x7f323c0b1f60 2024-03-29T10:45:44.818 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:44.833+0000 7f322affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:44.818 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:44.833+0000 7f322affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:45:44.818 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:44.833+0000 7f322affd640 10 monclient: dump: 2024-03-29T10:45:44.818 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:45:44.819 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:45:44.819 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:45:44.819 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:45:44.819 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:44.819 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:45:44.819 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:45:44.819 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:45:44.819 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:45:44.819 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:45:44.819 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:44.833+0000 7f3238d99640 4 set_mon_vals no callback set 2024-03-29T10:45:44.820 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:44.836+0000 7f322affd640 1 -- 172.21.15.88:0/1427160741 <== mon.1 v1:172.21.15.196:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4235629220 0 0) 0x7f322c01aed0 con 0x7f323c0b1f60 2024-03-29T10:45:44.820 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:44.837+0000 7f322affd640 1 -- 172.21.15.88:0/1427160741 <== mon.1 v1:172.21.15.196:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4257673582 0 0) 0x7f322c084bf0 con 0x7f323c0b1f60 2024-03-29T10:45:44.820 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:44.837+0000 7f322affd640 10 client.4402.objecter ms_dispatch 0x7f323c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:45:44.820 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:44.837+0000 7f322affd640 3 client.4402.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:45:44.820 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:44.837+0000 7f322affd640 3 client.4402.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:45:44.821 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:44.837+0000 7f322affd640 20 client.4402.objecter dump_active .. 0 homeless 2024-03-29T10:45:44.821 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:44.838+0000 7f3239d9b640 10 monclient: get_auth_request con 0x7f321c06db80 auth_method 0 2024-03-29T10:45:44.821 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:44.838+0000 7f322affd640 10 client.4402.objecter ms_handle_connect 0x7f321c06db80 2024-03-29T10:45:44.824 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:44.842+0000 7f322affd640 1 -- 172.21.15.88:0/1427160741 <== mon.1 v1:172.21.15.196:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f322c0504d0 con 0x7f323c0b1f60 2024-03-29T10:45:44.824 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:44.842+0000 7f322affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:45:44.824 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:44.842+0000 7f322affd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:45:45.145 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:45.162+0000 7f3240c15640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:45:45.145 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:45.162+0000 7f3240c15640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:45:45.145 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:45.162+0000 7f3240c15640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T10:45:45.145 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:45.162+0000 7f3240c15640 1 -- 172.21.15.88:0/1427160741 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f323c074f50 con 0x7f323c0b1f60 2024-03-29T10:45:45.147 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:45.164+0000 7f322affd640 1 -- 172.21.15.88:0/1427160741 <== mon.1 v1:172.21.15.196:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 344506374) 0x7f322c052400 con 0x7f323c0b1f60 2024-03-29T10:45:45.147 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:45.164+0000 7f322affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:45:45.147 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:45.164+0000 7f322affd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:45:45.147 INFO:teuthology.misc.health.smithi088.stdout: 2024-03-29T10:45:45.148 INFO:teuthology.misc.health.smithi088.stdout:{"epoch":10,"fsid":"869f5188-f92b-42dd-a46d-8c646e0388a9","created":"2024-03-29T10:40:40.285087+0000","modified":"2024-03-29T10:40:42.536230+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:40:42.536230+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a26c0c1c-8de0-438c-a3f6-3df10ea61574","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5de24915-8dec-49b6-8b16-f3170197c591","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ac054af3-d589-4565-9644-e6af75975c53","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3ec4f045-c714-4a22-bd1a-e07330b45ab3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"cddb9867-8e30-4951-9b13-3d1e78b9c45a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dc6d9911-785d-432f-8b53-56a1ccab302f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"77c88437-b587-496e-8b4d-1824ea8b5305","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ce963d96-5a8e-44c3-99de-c0dbebe234ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:45:45.151 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:45.168+0000 7f3240c15640 10 librados: watch_flush enter 2024-03-29T10:45:45.151 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:45.168+0000 7f3240c15640 10 librados: watch_flush exit 2024-03-29T10:45:45.151 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:45.168+0000 7f3240c15640 20 client.4402.objecter shutdown clearing up homeless session... 2024-03-29T10:45:45.151 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:45.168+0000 7f3240c15640 10 client.4402.objecter successfully canceled tick 2024-03-29T10:45:45.151 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:45.168+0000 7f3240c15640 1 -- 172.21.15.88:0/1427160741 >> v1:172.21.15.88:6816/1752861349 conn(0x7f321c06db80 legacy=0x7f321c070040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:45.151 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:45.168+0000 7f3240c15640 10 monclient: shutdown 2024-03-29T10:45:45.152 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:45.168+0000 7f3240c15640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:45:45.152 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:45.168+0000 7f3240c15640 1 -- 172.21.15.88:0/1427160741 >> v1:172.21.15.196:6789/0 conn(0x7f323c0b1f60 legacy=0x7f323c1e3d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:45.152 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:45.169+0000 7f3240c15640 1 -- 172.21.15.88:0/1427160741 shutdown_connections 2024-03-29T10:45:45.152 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:45.169+0000 7f3240c15640 1 -- 172.21.15.88:0/1427160741 >> 172.21.15.88:0/1427160741 conn(0x7f323c066a00 msgr2=0x7f323c0b5810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:45:45.152 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:45.169+0000 7f3240c15640 1 -- 172.21.15.88:0/1427160741 shutdown_connections 2024-03-29T10:45:45.152 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:45.169+0000 7f3240c15640 1 -- 172.21.15.88:0/1427160741 wait complete. 2024-03-29T10:45:45.153 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:45.169+0000 7f3240c15640 1 librados: shutdown 2024-03-29T10:45:45.167 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:45:51.976 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:45:52.122 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:52.139+0000 7fa4a8aab640 10 monclient: get_monmap_and_config 2024-03-29T10:45:52.122 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:52.139+0000 7fa4a8aab640 10 monclient: build_initial_monmap 2024-03-29T10:45:52.122 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:52.139+0000 7fa4a8aab640 10 monclient: monmap: 2024-03-29T10:45:52.122 INFO:teuthology.misc.health.smithi088.stderr:epoch 0 2024-03-29T10:45:52.122 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:45:52.122 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:45:52.140323+0000 2024-03-29T10:45:52.122 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:45:52.140323+0000 2024-03-29T10:45:52.122 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 0 (unknown) 2024-03-29T10:45:52.122 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:45:52.122 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-03-29T10:45:52.122 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-29T10:45:52.123 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-b 2024-03-29T10:45:52.123 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:45:52.123 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:52.140+0000 7fa4a8aab640 1 Processor -- start 2024-03-29T10:45:52.123 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:52.140+0000 7fa4a8aab640 1 -- start start 2024-03-29T10:45:52.123 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:52.140+0000 7fa4a8aab640 10 monclient: init 2024-03-29T10:45:52.124 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:52.141+0000 7fa4a8aab640 10 monclient: _reopen_session rank -1 2024-03-29T10:45:52.124 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:52.141+0000 7fa4a8aab640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:45:52.124 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:52.141+0000 7fa4a8aab640 10 monclient(hunting): picked mon.noname-c con 0x7fa49c02b4c0 addr v1:172.21.15.196:6789/0 2024-03-29T10:45:52.124 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:52.141+0000 7fa4a8aab640 10 monclient(hunting): picked mon.noname-a con 0x7fa49c00b8a0 addr v1:172.21.15.88:6789/0 2024-03-29T10:45:52.124 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:52.141+0000 7fa4a8aab640 10 monclient(hunting): picked mon.noname-b con 0x7fa49c02a5a0 addr v1:172.21.15.88:6790/0 2024-03-29T10:45:52.124 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:52.141+0000 7fa4a8aab640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa49c02c5c0 con 0x7fa49c00b8a0 2024-03-29T10:45:52.124 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:52.141+0000 7fa4a8aab640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa49c0c11a0 con 0x7fa49c02b4c0 2024-03-29T10:45:52.124 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:52.141+0000 7fa4a8aab640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa49c02c830 con 0x7fa49c02a5a0 2024-03-29T10:45:52.124 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:52.141+0000 7fa4a8aab640 10 monclient(hunting): _renew_subs 2024-03-29T10:45:52.124 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:52.141+0000 7fa4a8aab640 10 monclient(hunting): authenticate will time out at 1074.377358s 2024-03-29T10:45:52.127 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:52.145+0000 7fa4a37fe640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7fa49c02b4c0 0x7fa49c006890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.88:52007/0 (socket says 172.21.15.88:52007) 2024-03-29T10:45:52.127 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:52.145+0000 7fa4a37fe640 1 -- 172.21.15.88:0/3843715087 learned_addr learned my addr 172.21.15.88:0/3843715087 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-29T10:45:52.128 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:52.145+0000 7fa4a3fff640 1 --1- 172.21.15.88:0/3843715087 >> v1:172.21.15.88:6790/0 conn(0x7fa49c02a5a0 0x7fa49c0c6d40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6790/0 says I am v1:172.21.15.88:37205/0 (socket says 172.21.15.88:37205) 2024-03-29T10:45:52.128 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:52.146+0000 7fa4a27fc640 1 -- 172.21.15.88:0/3843715087 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1878770284 0 0) 0x7fa49c02c5c0 con 0x7fa49c00b8a0 2024-03-29T10:45:52.128 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:52.146+0000 7fa4a27fc640 10 monclient(hunting): _negotiate 2024-03-29T10:45:52.128 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:52.146+0000 7fa4a27fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:45:52.128 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:52.146+0000 7fa4a27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:45:52.128 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:52.146+0000 7fa4a27fc640 10 monclient(hunting): my global_id is 4410 2024-03-29T10:45:52.128 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:52.146+0000 7fa4a27fc640 1 -- 172.21.15.88:0/3843715087 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa494003520 con 0x7fa49c00b8a0 2024-03-29T10:45:52.132 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:52.149+0000 7fa4a27fc640 1 -- 172.21.15.88:0/3843715087 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3587158698 0 0) 0x7fa49c02c830 con 0x7fa49c02a5a0 2024-03-29T10:45:52.132 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:52.149+0000 7fa4a27fc640 10 monclient(hunting): _negotiate 2024-03-29T10:45:52.132 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:52.149+0000 7fa4a27fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:45:52.132 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:52.149+0000 7fa4a27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:45:52.133 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:52.149+0000 7fa4a27fc640 10 monclient(hunting): my global_id is 4436 2024-03-29T10:45:52.133 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:52.149+0000 7fa4a27fc640 1 -- 172.21.15.88:0/3843715087 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa49c02c5c0 con 0x7fa49c02a5a0 2024-03-29T10:45:52.133 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:52.149+0000 7fa4a27fc640 1 -- 172.21.15.88:0/3843715087 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2699016193 0 0) 0x7fa49c0c11a0 con 0x7fa49c02b4c0 2024-03-29T10:45:52.133 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:52.149+0000 7fa4a27fc640 10 monclient(hunting): _negotiate 2024-03-29T10:45:52.133 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:52.149+0000 7fa4a27fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:45:52.133 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:52.149+0000 7fa4a27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:45:52.133 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:52.149+0000 7fa4a27fc640 10 monclient(hunting): my global_id is 4405 2024-03-29T10:45:52.133 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:52.149+0000 7fa4a27fc640 1 -- 172.21.15.88:0/3843715087 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa49c02c830 con 0x7fa49c02b4c0 2024-03-29T10:45:52.133 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:52.149+0000 7fa4a27fc640 1 -- 172.21.15.88:0/3843715087 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2308090293 0 0) 0x7fa494003520 con 0x7fa49c00b8a0 2024-03-29T10:45:52.133 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:52.149+0000 7fa4a27fc640 1 -- 172.21.15.88:0/3843715087 >> v1:172.21.15.88:6790/0 conn(0x7fa49c02a5a0 legacy=0x7fa49c0c6d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:52.133 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:52.149+0000 7fa4a27fc640 1 -- 172.21.15.88:0/3843715087 >> v1:172.21.15.196:6789/0 conn(0x7fa49c02b4c0 legacy=0x7fa49c006890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:52.133 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:52.149+0000 7fa4a27fc640 10 monclient: _finish_hunting 0 2024-03-29T10:45:52.133 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:52.149+0000 7fa4a27fc640 1 monclient: found mon.noname-a 2024-03-29T10:45:52.133 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:52.149+0000 7fa4a27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:45:52.134 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:52.149+0000 7fa4a27fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.88:6789/0 2024-03-29T10:45:52.134 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:52.149+0000 7fa4a27fc640 1 -- 172.21.15.88:0/3843715087 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa49c02afa0 con 0x7fa49c00b8a0 2024-03-29T10:45:52.134 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:52.149+0000 7fa4a27fc640 10 monclient: _finish_auth 0 2024-03-29T10:45:52.134 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:52.149+0000 7fa4a27fc640 10 monclient: _check_auth_tickets 2024-03-29T10:45:52.134 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:52.149+0000 7fa4a27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:45:52.134 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:52.149+0000 7fa4a8aab640 5 monclient: authenticate success, global_id 4410 2024-03-29T10:45:52.134 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:52.149+0000 7fa4a8aab640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:45:52.134 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:52.149+0000 7fa4a27fc640 1 -- 172.21.15.88:0/3843715087 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7fa490001610 con 0x7fa49c00b8a0 2024-03-29T10:45:52.134 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:52.149+0000 7fa4a27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:52.134 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:52.149+0000 7fa4a27fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:45:52.134 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:52.149+0000 7fa4a27fc640 10 monclient: dump: 2024-03-29T10:45:52.134 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:45:52.134 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:45:52.134 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:45:52.135 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:45:52.135 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:52.135 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:45:52.135 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:45:52.135 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:45:52.135 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:45:52.135 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:45:52.135 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:52.149+0000 7fa4a8aab640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:45:52.135 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:52.150+0000 7fa4a27fc640 1 -- 172.21.15.88:0/3843715087 <== mon.0 v1:172.21.15.88:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa490001830 con 0x7fa49c00b8a0 2024-03-29T10:45:52.135 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:52.150+0000 7fa4a27fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:45:52.135 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:52.150+0000 7fa4a8aab640 10 monclient: get_monmap_and_config success 2024-03-29T10:45:52.135 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:52.150+0000 7fa4a8aab640 4 set_mon_vals no callback set 2024-03-29T10:45:52.136 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:52.150+0000 7fa4a27fc640 1 -- 172.21.15.88:0/3843715087 <== mon.0 v1:172.21.15.88:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7fa490001bc0 con 0x7fa49c00b8a0 2024-03-29T10:45:52.136 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:52.150+0000 7fa4a27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:52.137 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:52.150+0000 7fa4a27fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:45:52.137 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:52.150+0000 7fa4a27fc640 10 monclient: dump: 2024-03-29T10:45:52.137 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:45:52.137 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:45:52.137 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:45:52.137 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:45:52.137 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:52.137 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:45:52.137 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:45:52.137 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:45:52.137 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:45:52.137 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:45:52.137 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:52.150+0000 7fa4a8aab640 10 monclient: shutdown 2024-03-29T10:45:52.137 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:52.150+0000 7fa4a8aab640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:45:52.138 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:52.150+0000 7fa4a8aab640 1 -- 172.21.15.88:0/3843715087 >> v1:172.21.15.88:6789/0 conn(0x7fa49c00b8a0 legacy=0x7fa49c0c0a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:52.138 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:52.151+0000 7fa4a8aab640 1 -- 172.21.15.88:0/3843715087 shutdown_connections 2024-03-29T10:45:52.138 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:52.151+0000 7fa4a8aab640 1 -- 172.21.15.88:0/3843715087 >> 172.21.15.88:0/3843715087 conn(0x7fa49c0058f0 msgr2=0x7fa49c005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:45:52.138 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:52.152+0000 7fa4a8aab640 1 -- 172.21.15.88:0/3843715087 shutdown_connections 2024-03-29T10:45:52.138 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:52.152+0000 7fa4a8aab640 1 -- 172.21.15.88:0/3843715087 wait complete. 2024-03-29T10:45:52.138 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:52.153+0000 7fa4a8aab640 10 monclient: build_initial_monmap 2024-03-29T10:45:52.138 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:52.153+0000 7fa4a8aab640 10 monclient: monmap: 2024-03-29T10:45:52.138 INFO:teuthology.misc.health.smithi088.stderr:epoch 0 2024-03-29T10:45:52.138 INFO:teuthology.misc.health.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:45:52.138 INFO:teuthology.misc.health.smithi088.stderr:last_changed 0.000000 2024-03-29T10:45:52.139 INFO:teuthology.misc.health.smithi088.stderr:created 0.000000 2024-03-29T10:45:52.139 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 0 (unknown) 2024-03-29T10:45:52.139 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:45:52.139 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-03-29T10:45:52.139 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-29T10:45:52.139 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-c 2024-03-29T10:45:52.139 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:45:52.139 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:52.154+0000 7fa4a8aab640 1 librados: starting msgr at 2024-03-29T10:45:52.139 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:52.154+0000 7fa4a8aab640 1 librados: starting objecter 2024-03-29T10:45:52.139 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:52.154+0000 7fa4a8aab640 1 Processor -- start 2024-03-29T10:45:52.142 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:52.154+0000 7fa4a8aab640 1 -- start start 2024-03-29T10:45:52.142 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:52.154+0000 7fa4a8aab640 1 librados: setting wanted keys 2024-03-29T10:45:52.142 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:52.154+0000 7fa4a8aab640 1 librados: calling monclient init 2024-03-29T10:45:52.142 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:52.154+0000 7fa4a8aab640 10 monclient: init 2024-03-29T10:45:52.142 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:52.155+0000 7fa4a8aab640 10 monclient: _reopen_session rank -1 2024-03-29T10:45:52.142 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:52.155+0000 7fa4a8aab640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:45:52.142 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:52.155+0000 7fa4a8aab640 10 monclient(hunting): picked mon.noname-b con 0x7fa49c00b8a0 addr v1:172.21.15.196:6789/0 2024-03-29T10:45:52.142 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:52.155+0000 7fa4a8aab640 10 monclient(hunting): picked mon.noname-a con 0x7fa49c02a5a0 addr v1:172.21.15.88:6789/0 2024-03-29T10:45:52.142 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:52.155+0000 7fa4a8aab640 10 monclient(hunting): picked mon.noname-c con 0x7fa49c02b4c0 addr v1:172.21.15.88:6790/0 2024-03-29T10:45:52.142 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:52.155+0000 7fa4a8aab640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa49c0d5710 con 0x7fa49c02a5a0 2024-03-29T10:45:52.142 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:52.155+0000 7fa4a8aab640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa49c0d5430 con 0x7fa49c00b8a0 2024-03-29T10:45:52.142 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:52.155+0000 7fa4a8aab640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa49c0d5ba0 con 0x7fa49c02b4c0 2024-03-29T10:45:52.143 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:52.155+0000 7fa4a8aab640 10 monclient(hunting): _renew_subs 2024-03-29T10:45:52.143 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:52.155+0000 7fa4a8aab640 10 monclient(hunting): authenticate will time out at 1074.391154s 2024-03-29T10:45:52.143 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:52.159+0000 7fa4a2ffd640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fa49c02a5a0 0x7fa49c02d1a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:56579/0 (socket says 172.21.15.88:56579) 2024-03-29T10:45:52.143 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:52.159+0000 7fa4a2ffd640 1 -- 172.21.15.88:0/3064215702 learned_addr learned my addr 172.21.15.88:0/3064215702 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-29T10:45:52.143 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:52.159+0000 7fa48ffff640 10 client.?.objecter ms_handle_connect 0x7fa49c02a5a0 2024-03-29T10:45:52.143 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:52.159+0000 7fa48ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:45:52.143 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:52.159+0000 7fa48ffff640 1 -- 172.21.15.88:0/3064215702 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3086059077 0 0) 0x7fa49c0d5710 con 0x7fa49c02a5a0 2024-03-29T10:45:52.143 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:52.159+0000 7fa48ffff640 10 monclient(hunting): _negotiate 2024-03-29T10:45:52.143 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:52.159+0000 7fa48ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:45:52.144 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:52.159+0000 7fa48ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:45:52.144 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:52.159+0000 7fa48ffff640 10 monclient(hunting): my global_id is 4413 2024-03-29T10:45:52.144 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:52.159+0000 7fa48ffff640 1 -- 172.21.15.88:0/3064215702 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa488004eb0 con 0x7fa49c02a5a0 2024-03-29T10:45:52.144 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:52.161+0000 7fa48ffff640 10 client.?.objecter ms_handle_connect 0x7fa49c02b4c0 2024-03-29T10:45:52.144 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:52.161+0000 7fa48ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:45:52.144 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:52.161+0000 7fa48ffff640 1 -- 172.21.15.88:0/3064215702 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1505641262 0 0) 0x7fa488004eb0 con 0x7fa49c02a5a0 2024-03-29T10:45:52.144 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:52.161+0000 7fa48ffff640 1 -- 172.21.15.88:0/3064215702 --> v1:172.21.15.88:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa49c0d5710 con 0x7fa49c02a5a0 2024-03-29T10:45:52.144 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:52.161+0000 7fa48ffff640 1 -- 172.21.15.88:0/3064215702 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7fa490001c80 con 0x7fa49c02a5a0 2024-03-29T10:45:52.144 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:52.161+0000 7fa48ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:52.144 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:52.161+0000 7fa48ffff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:45:52.144 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:52.161+0000 7fa48ffff640 10 monclient(hunting): dump: 2024-03-29T10:45:52.145 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:45:52.145 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:45:52.145 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:45:52.145 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:45:52.145 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:52.145 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:45:52.145 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:45:52.145 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:45:52.145 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:45:52.145 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:45:52.145 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:52.161+0000 7fa48ffff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:45:52.146 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:52.163+0000 7fa48ffff640 1 -- 172.21.15.88:0/3064215702 <== mon.0 v1:172.21.15.88:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3675325527 0 0) 0x7fa49c0d5710 con 0x7fa49c02a5a0 2024-03-29T10:45:52.146 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:52.163+0000 7fa48ffff640 1 -- 172.21.15.88:0/3064215702 >> v1:172.21.15.88:6790/0 conn(0x7fa49c02b4c0 legacy=0x7fa49c16c810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:52.146 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:52.163+0000 7fa48ffff640 1 -- 172.21.15.88:0/3064215702 >> v1:172.21.15.196:6789/0 conn(0x7fa49c00b8a0 legacy=0x7fa49c0b0e50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:45:52.147 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:52.165+0000 7fa48ffff640 10 monclient: _finish_hunting 0 2024-03-29T10:45:52.147 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:52.165+0000 7fa48ffff640 1 monclient: found mon.a 2024-03-29T10:45:52.147 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:52.165+0000 7fa48ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:45:52.147 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:52.165+0000 7fa48ffff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-03-29T10:45:52.148 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:52.165+0000 7fa48ffff640 1 -- 172.21.15.88:0/3064215702 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa49c0d6280 con 0x7fa49c02a5a0 2024-03-29T10:45:52.148 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:52.165+0000 7fa48ffff640 10 monclient: _finish_auth 0 2024-03-29T10:45:52.148 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:52.165+0000 7fa48ffff640 10 monclient: _check_auth_tickets 2024-03-29T10:45:52.148 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:52.165+0000 7fa48ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:45:52.148 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:52.166+0000 7fa48ffff640 1 -- 172.21.15.88:0/3064215702 <== mon.0 v1:172.21.15.88:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa4900038a0 con 0x7fa49c02a5a0 2024-03-29T10:45:52.149 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:52.166+0000 7fa48ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:45:52.149 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:52.166+0000 7fa4a1ffb640 4 set_mon_vals no callback set 2024-03-29T10:45:52.149 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:52.167+0000 7fa48ffff640 1 -- 172.21.15.88:0/3064215702 <== mon.0 v1:172.21.15.88:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7fa490003cb0 con 0x7fa49c02a5a0 2024-03-29T10:45:52.149 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:52.167+0000 7fa48ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:52.149 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:52.167+0000 7fa48ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:45:52.149 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:52.167+0000 7fa48ffff640 10 monclient: dump: 2024-03-29T10:45:52.149 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:45:52.149 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:45:52.149 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:45:52.149 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:45:52.150 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:52.150 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:45:52.150 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:45:52.150 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:45:52.150 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:45:52.150 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:45:52.150 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:52.168+0000 7fa4a8aab640 5 monclient: authenticate success, global_id 4413 2024-03-29T10:45:52.151 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:52.168+0000 7fa4a8aab640 10 monclient: _renew_subs 2024-03-29T10:45:52.151 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:52.168+0000 7fa4a8aab640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-03-29T10:45:52.151 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:52.168+0000 7fa4a8aab640 1 -- 172.21.15.88:0/3064215702 --> v1:172.21.15.88:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa49c0d6470 con 0x7fa49c02a5a0 2024-03-29T10:45:52.151 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:52.168+0000 7fa4a8aab640 10 client.4413.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:45:52.151 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:52.168+0000 7fa4a8aab640 10 monclient: _renew_subs 2024-03-29T10:45:52.151 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:52.168+0000 7fa4a8aab640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-03-29T10:45:52.151 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:52.168+0000 7fa4a8aab640 1 -- 172.21.15.88:0/3064215702 --> v1:172.21.15.88:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa49c0d69e0 con 0x7fa49c02a5a0 2024-03-29T10:45:52.151 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:52.168+0000 7fa4a8aab640 1 librados: init done 2024-03-29T10:45:52.153 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:52.171+0000 7fa48ffff640 1 -- 172.21.15.88:0/3064215702 <== mon.0 v1:172.21.15.88:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4235629220 0 0) 0x7fa490003e10 con 0x7fa49c02a5a0 2024-03-29T10:45:52.155 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:52.172+0000 7fa4a37fe640 10 monclient: get_auth_request con 0x7fa48806dc30 auth_method 0 2024-03-29T10:45:52.156 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:52.173+0000 7fa48ffff640 1 -- 172.21.15.88:0/3064215702 <== mon.0 v1:172.21.15.88:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4257673582 0 0) 0x7fa490084c20 con 0x7fa49c02a5a0 2024-03-29T10:45:52.156 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:52.173+0000 7fa48ffff640 10 client.4413.objecter ms_dispatch 0x7fa4a4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:45:52.156 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:52.173+0000 7fa48ffff640 3 client.4413.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:45:52.156 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:52.173+0000 7fa48ffff640 3 client.4413.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:45:52.156 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:52.173+0000 7fa48ffff640 20 client.4413.objecter dump_active .. 0 homeless 2024-03-29T10:45:52.156 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:52.173+0000 7fa48ffff640 10 client.4413.objecter ms_handle_connect 0x7fa48806dc30 2024-03-29T10:45:52.156 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:52.174+0000 7fa4a8aab640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:45:52.157 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:52.174+0000 7fa4a8aab640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:45:52.157 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:52.174+0000 7fa4a8aab640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-03-29T10:45:52.157 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:52.174+0000 7fa4a8aab640 1 -- 172.21.15.88:0/3064215702 --> v1:172.21.15.88:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa49c0d6600 con 0x7fa49c02a5a0 2024-03-29T10:45:52.164 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:52.182+0000 7fa48ffff640 1 -- 172.21.15.88:0/3064215702 <== mon.0 v1:172.21.15.88:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa490050500 con 0x7fa49c02a5a0 2024-03-29T10:45:52.164 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:52.182+0000 7fa48ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:45:52.165 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:52.182+0000 7fa48ffff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:45:52.489 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:52.506+0000 7fa4a8aab640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:45:52.489 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:52.506+0000 7fa4a8aab640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:45:52.490 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:52.506+0000 7fa4a8aab640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-03-29T10:45:52.490 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:52.506+0000 7fa4a8aab640 1 -- 172.21.15.88:0/3064215702 --> v1:172.21.15.88:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa49c004cd0 con 0x7fa49c02a5a0 2024-03-29T10:45:52.490 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:52.507+0000 7fa48ffff640 1 -- 172.21.15.88:0/3064215702 <== mon.0 v1:172.21.15.88:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 344506374) 0x7fa490052430 con 0x7fa49c02a5a0 2024-03-29T10:45:52.490 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:52.507+0000 7fa48ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:45:52.490 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:52.507+0000 7fa48ffff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:45:52.492 INFO:teuthology.misc.health.smithi088.stdout: 2024-03-29T10:45:52.492 INFO:teuthology.misc.health.smithi088.stdout:{"epoch":10,"fsid":"869f5188-f92b-42dd-a46d-8c646e0388a9","created":"2024-03-29T10:40:40.285087+0000","modified":"2024-03-29T10:40:42.536230+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:40:42.536230+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a26c0c1c-8de0-438c-a3f6-3df10ea61574","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5de24915-8dec-49b6-8b16-f3170197c591","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ac054af3-d589-4565-9644-e6af75975c53","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3ec4f045-c714-4a22-bd1a-e07330b45ab3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"cddb9867-8e30-4951-9b13-3d1e78b9c45a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dc6d9911-785d-432f-8b53-56a1ccab302f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"77c88437-b587-496e-8b4d-1824ea8b5305","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ce963d96-5a8e-44c3-99de-c0dbebe234ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:45:52.495 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:52.512+0000 7fa4a8aab640 10 librados: watch_flush enter 2024-03-29T10:45:52.495 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:52.512+0000 7fa4a8aab640 10 librados: watch_flush exit 2024-03-29T10:45:52.495 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:52.512+0000 7fa4a8aab640 20 client.4413.objecter shutdown clearing up homeless session... 2024-03-29T10:45:52.495 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:52.512+0000 7fa4a8aab640 10 client.4413.objecter successfully canceled tick 2024-03-29T10:45:52.497 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:52.514+0000 7fa4a8aab640 1 -- 172.21.15.88:0/3064215702 >> v1:172.21.15.88:6816/1752861349 conn(0x7fa48806dc30 legacy=0x7fa4880700f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:52.497 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:52.514+0000 7fa4a8aab640 10 monclient: shutdown 2024-03-29T10:45:52.498 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:52.514+0000 7fa4a8aab640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:45:52.498 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:52.514+0000 7fa4a8aab640 1 -- 172.21.15.88:0/3064215702 >> v1:172.21.15.88:6789/0 conn(0x7fa49c02a5a0 legacy=0x7fa49c02d1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:52.498 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:52.515+0000 7fa4a8aab640 1 -- 172.21.15.88:0/3064215702 shutdown_connections 2024-03-29T10:45:52.498 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:52.515+0000 7fa4a8aab640 1 -- 172.21.15.88:0/3064215702 >> 172.21.15.88:0/3064215702 conn(0x7fa49c0058f0 msgr2=0x7fa49c0382d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:45:52.498 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:52.515+0000 7fa4a8aab640 1 -- 172.21.15.88:0/3064215702 shutdown_connections 2024-03-29T10:45:52.498 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:52.515+0000 7fa4a8aab640 1 -- 172.21.15.88:0/3064215702 wait complete. 2024-03-29T10:45:52.498 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:52.515+0000 7fa4a8aab640 1 librados: shutdown 2024-03-29T10:45:52.514 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:45:59.320 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:45:59.468 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:59.485+0000 7fc1d5fd6640 10 monclient: get_monmap_and_config 2024-03-29T10:45:59.468 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:59.485+0000 7fc1d5fd6640 10 monclient: build_initial_monmap 2024-03-29T10:45:59.468 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:59.485+0000 7fc1d5fd6640 10 monclient: monmap: 2024-03-29T10:45:59.468 INFO:teuthology.misc.health.smithi088.stderr:epoch 0 2024-03-29T10:45:59.468 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:45:59.468 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:45:59.486517+0000 2024-03-29T10:45:59.468 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:45:59.486517+0000 2024-03-29T10:45:59.468 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 0 (unknown) 2024-03-29T10:45:59.468 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:45:59.469 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-03-29T10:45:59.469 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-29T10:45:59.469 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-b 2024-03-29T10:45:59.469 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:45:59.469 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:59.486+0000 7fc1d5fd6640 1 Processor -- start 2024-03-29T10:45:59.469 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:59.486+0000 7fc1d5fd6640 1 -- start start 2024-03-29T10:45:59.469 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:59.486+0000 7fc1d5fd6640 10 monclient: init 2024-03-29T10:45:59.472 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:59.487+0000 7fc1d5fd6640 10 monclient: _reopen_session rank -1 2024-03-29T10:45:59.473 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:59.487+0000 7fc1d5fd6640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:45:59.473 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:59.487+0000 7fc1d5fd6640 10 monclient(hunting): picked mon.noname-a con 0x7fc1c8006da0 addr v1:172.21.15.88:6789/0 2024-03-29T10:45:59.473 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:59.487+0000 7fc1d5fd6640 10 monclient(hunting): picked mon.noname-b con 0x7fc1c80cc330 addr v1:172.21.15.88:6790/0 2024-03-29T10:45:59.473 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:59.487+0000 7fc1d5fd6640 10 monclient(hunting): picked mon.noname-c con 0x7fc1c80c9450 addr v1:172.21.15.196:6789/0 2024-03-29T10:45:59.473 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:59.487+0000 7fc1d5fd6640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1c8008490 con 0x7fc1c8006da0 2024-03-29T10:45:59.473 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:59.487+0000 7fc1d5fd6640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1c802be50 con 0x7fc1c80c9450 2024-03-29T10:45:59.473 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:59.487+0000 7fc1d5fd6640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1c802c130 con 0x7fc1c80cc330 2024-03-29T10:45:59.473 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:59.487+0000 7fc1d5fd6640 10 monclient(hunting): _renew_subs 2024-03-29T10:45:59.473 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:59.487+0000 7fc1d5fd6640 10 monclient(hunting): authenticate will time out at 1081.723611s 2024-03-29T10:45:59.476 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:59.493+0000 7fc1cffff640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7fc1c80c9450 0x7fc1c80d74e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.88:50843/0 (socket says 172.21.15.88:50843) 2024-03-29T10:45:59.476 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:59.493+0000 7fc1cffff640 1 -- 172.21.15.88:0/1316534140 learned_addr learned my addr 172.21.15.88:0/1316534140 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-29T10:45:59.477 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:59.494+0000 7fc1ce7fc640 1 -- 172.21.15.88:0/1316534140 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 958301097 0 0) 0x7fc1c802be50 con 0x7fc1c80c9450 2024-03-29T10:45:59.477 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:59.494+0000 7fc1ce7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:45:59.477 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:59.494+0000 7fc1ce7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:45:59.477 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:59.494+0000 7fc1ce7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:45:59.477 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:59.494+0000 7fc1ce7fc640 10 monclient(hunting): my global_id is 4408 2024-03-29T10:45:59.477 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:59.494+0000 7fc1ce7fc640 1 -- 172.21.15.88:0/1316534140 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc1c0003520 con 0x7fc1c80c9450 2024-03-29T10:45:59.477 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:59.494+0000 7fc1ce7fc640 1 -- 172.21.15.88:0/1316534140 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2872910797 0 0) 0x7fc1c8008490 con 0x7fc1c8006da0 2024-03-29T10:45:59.477 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:59.494+0000 7fc1ce7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:45:59.477 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:59.494+0000 7fc1ce7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:45:59.477 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:59.494+0000 7fc1ce7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:45:59.477 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:59.494+0000 7fc1ce7fc640 10 monclient(hunting): my global_id is 4416 2024-03-29T10:45:59.478 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:59.494+0000 7fc1ce7fc640 1 -- 172.21.15.88:0/1316534140 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc1c0003840 con 0x7fc1c8006da0 2024-03-29T10:45:59.479 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:59.494+0000 7fc1ce7fc640 1 -- 172.21.15.88:0/1316534140 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4162993695 0 0) 0x7fc1c0003840 con 0x7fc1c8006da0 2024-03-29T10:45:59.479 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:59.494+0000 7fc1ce7fc640 1 -- 172.21.15.88:0/1316534140 >> v1:172.21.15.88:6790/0 conn(0x7fc1c80cc330 legacy=0x7fc1c80c8d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:45:59.479 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:59.494+0000 7fc1ce7fc640 1 -- 172.21.15.88:0/1316534140 >> v1:172.21.15.196:6789/0 conn(0x7fc1c80c9450 legacy=0x7fc1c80d74e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:59.479 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:59.494+0000 7fc1ce7fc640 10 monclient: _finish_hunting 0 2024-03-29T10:45:59.480 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:59.494+0000 7fc1ce7fc640 1 monclient: found mon.noname-a 2024-03-29T10:45:59.480 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:59.494+0000 7fc1ce7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:45:59.480 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:59.494+0000 7fc1ce7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.88:6789/0 2024-03-29T10:45:59.480 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:59.494+0000 7fc1ce7fc640 1 -- 172.21.15.88:0/1316534140 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1c802c410 con 0x7fc1c8006da0 2024-03-29T10:45:59.480 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:59.494+0000 7fc1ce7fc640 10 monclient: _finish_auth 0 2024-03-29T10:45:59.480 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:59.494+0000 7fc1ce7fc640 10 monclient: _check_auth_tickets 2024-03-29T10:45:59.480 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:59.494+0000 7fc1ce7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:45:59.480 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:59.494+0000 7fc1d5fd6640 5 monclient: authenticate success, global_id 4416 2024-03-29T10:45:59.480 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:59.494+0000 7fc1d5fd6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:45:59.480 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:59.495+0000 7fc1ce7fc640 1 -- 172.21.15.88:0/1316534140 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7fc1d004f730 con 0x7fc1c8006da0 2024-03-29T10:45:59.480 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:59.495+0000 7fc1ce7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:59.480 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:59.495+0000 7fc1ce7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:45:59.480 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:59.495+0000 7fc1ce7fc640 10 monclient: dump: 2024-03-29T10:45:59.480 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:45:59.481 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:45:59.481 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:45:59.481 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:45:59.481 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:59.481 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:45:59.481 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:45:59.481 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:45:59.481 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:45:59.481 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:45:59.481 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:59.495+0000 7fc1d5fd6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:45:59.481 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:59.495+0000 7fc1ce7fc640 1 -- 172.21.15.88:0/1316534140 <== mon.0 v1:172.21.15.88:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc1d00008d0 con 0x7fc1c8006da0 2024-03-29T10:45:59.481 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:59.495+0000 7fc1ce7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:45:59.481 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:59.495+0000 7fc1d5fd6640 10 monclient: get_monmap_and_config success 2024-03-29T10:45:59.481 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:59.495+0000 7fc1d5fd6640 4 set_mon_vals no callback set 2024-03-29T10:45:59.482 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:59.496+0000 7fc1d5fd6640 10 monclient: shutdown 2024-03-29T10:45:59.482 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:59.496+0000 7fc1d5fd6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:45:59.482 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:59.496+0000 7fc1d5fd6640 1 -- 172.21.15.88:0/1316534140 >> v1:172.21.15.88:6789/0 conn(0x7fc1c8006da0 legacy=0x7fc1c80cbc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:59.482 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:59.499+0000 7fc1d5fd6640 1 -- 172.21.15.88:0/1316534140 shutdown_connections 2024-03-29T10:45:59.482 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:59.499+0000 7fc1d5fd6640 1 -- 172.21.15.88:0/1316534140 >> 172.21.15.88:0/1316534140 conn(0x7fc1c8005c20 msgr2=0x7fc1c8006030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:45:59.485 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:59.503+0000 7fc1d5fd6640 1 -- 172.21.15.88:0/1316534140 shutdown_connections 2024-03-29T10:45:59.485 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:59.503+0000 7fc1d5fd6640 1 -- 172.21.15.88:0/1316534140 wait complete. 2024-03-29T10:45:59.487 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:59.504+0000 7fc1d5fd6640 10 monclient: build_initial_monmap 2024-03-29T10:45:59.487 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:59.504+0000 7fc1d5fd6640 10 monclient: monmap: 2024-03-29T10:45:59.487 INFO:teuthology.misc.health.smithi088.stderr:epoch 0 2024-03-29T10:45:59.487 INFO:teuthology.misc.health.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:45:59.487 INFO:teuthology.misc.health.smithi088.stderr:last_changed 0.000000 2024-03-29T10:45:59.488 INFO:teuthology.misc.health.smithi088.stderr:created 0.000000 2024-03-29T10:45:59.488 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 0 (unknown) 2024-03-29T10:45:59.488 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:45:59.488 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-03-29T10:45:59.488 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-29T10:45:59.488 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-c 2024-03-29T10:45:59.488 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:45:59.488 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:59.505+0000 7fc1d5fd6640 1 librados: starting msgr at 2024-03-29T10:45:59.488 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:59.505+0000 7fc1d5fd6640 1 librados: starting objecter 2024-03-29T10:45:59.488 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:59.505+0000 7fc1d5fd6640 1 Processor -- start 2024-03-29T10:45:59.491 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:59.506+0000 7fc1d5fd6640 1 -- start start 2024-03-29T10:45:59.491 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:59.506+0000 7fc1d5fd6640 1 librados: setting wanted keys 2024-03-29T10:45:59.491 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:59.506+0000 7fc1d5fd6640 1 librados: calling monclient init 2024-03-29T10:45:59.491 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:59.506+0000 7fc1d5fd6640 10 monclient: init 2024-03-29T10:45:59.491 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:59.507+0000 7fc1d5fd6640 10 monclient: _reopen_session rank -1 2024-03-29T10:45:59.491 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:59.507+0000 7fc1d5fd6640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:45:59.491 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:59.507+0000 7fc1d5fd6640 10 monclient(hunting): picked mon.noname-a con 0x7fc1c8006da0 addr v1:172.21.15.88:6789/0 2024-03-29T10:45:59.491 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:59.507+0000 7fc1d5fd6640 10 monclient(hunting): picked mon.noname-c con 0x7fc1c80c9450 addr v1:172.21.15.88:6790/0 2024-03-29T10:45:59.491 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:59.507+0000 7fc1d5fd6640 10 monclient(hunting): picked mon.noname-b con 0x7fc1c80cc330 addr v1:172.21.15.196:6789/0 2024-03-29T10:45:59.491 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:59.507+0000 7fc1d5fd6640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1c80cb0a0 con 0x7fc1c8006da0 2024-03-29T10:45:59.491 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:59.507+0000 7fc1d5fd6640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1c81614e0 con 0x7fc1c80cc330 2024-03-29T10:45:59.492 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:59.507+0000 7fc1d5fd6640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1c80c4270 con 0x7fc1c80c9450 2024-03-29T10:45:59.492 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:59.507+0000 7fc1d5fd6640 10 monclient(hunting): _renew_subs 2024-03-29T10:45:59.492 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:59.507+0000 7fc1d5fd6640 10 monclient(hunting): authenticate will time out at 1081.743013s 2024-03-29T10:45:59.498 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:59.510+0000 7fc1cf7fe640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fc1c8006da0 0x7fc1c8160dd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:35017/0 (socket says 172.21.15.88:35017) 2024-03-29T10:45:59.498 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:59.510+0000 7fc1cf7fe640 1 -- 172.21.15.88:0/3273302681 learned_addr learned my addr 172.21.15.88:0/3273302681 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-29T10:45:59.498 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:59.510+0000 7fc1bbfff640 10 client.?.objecter ms_handle_connect 0x7fc1c8006da0 2024-03-29T10:45:59.498 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:59.510+0000 7fc1bbfff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:45:59.498 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:59.510+0000 7fc1bbfff640 1 -- 172.21.15.88:0/3273302681 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2750305944 0 0) 0x7fc1c80cb0a0 con 0x7fc1c8006da0 2024-03-29T10:45:59.498 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:59.510+0000 7fc1bbfff640 10 monclient(hunting): _negotiate 2024-03-29T10:45:59.498 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:59.510+0000 7fc1bbfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:45:59.498 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:59.510+0000 7fc1bbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:45:59.499 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:59.510+0000 7fc1bbfff640 10 monclient(hunting): my global_id is 4419 2024-03-29T10:45:59.499 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:59.510+0000 7fc1bbfff640 1 -- 172.21.15.88:0/3273302681 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc1b4004eb0 con 0x7fc1c8006da0 2024-03-29T10:45:59.499 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:59.511+0000 7fc1bbfff640 1 -- 172.21.15.88:0/3273302681 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2966018653 0 0) 0x7fc1b4004eb0 con 0x7fc1c8006da0 2024-03-29T10:45:59.499 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:59.511+0000 7fc1bbfff640 1 -- 172.21.15.88:0/3273302681 --> v1:172.21.15.88:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc1c80cb0a0 con 0x7fc1c8006da0 2024-03-29T10:45:59.499 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:59.511+0000 7fc1bbfff640 10 client.?.objecter ms_handle_connect 0x7fc1c80c9450 2024-03-29T10:45:59.499 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:59.511+0000 7fc1bbfff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:45:59.499 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:59.511+0000 7fc1bbfff640 1 -- 172.21.15.88:0/3273302681 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7fc1d0075710 con 0x7fc1c8006da0 2024-03-29T10:45:59.499 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:59.511+0000 7fc1bbfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:59.499 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:59.511+0000 7fc1bbfff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:45:59.499 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:59.511+0000 7fc1bbfff640 10 monclient(hunting): dump: 2024-03-29T10:45:59.499 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:45:59.499 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:45:59.499 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:45:59.500 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:45:59.500 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:59.500 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:45:59.500 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:45:59.500 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:45:59.500 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:45:59.500 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:45:59.500 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:59.511+0000 7fc1bbfff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:45:59.500 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:59.511+0000 7fc1bbfff640 1 -- 172.21.15.88:0/3273302681 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2398920375 0 0) 0x7fc1c80c4270 con 0x7fc1c80c9450 2024-03-29T10:45:59.500 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:59.511+0000 7fc1bbfff640 10 monclient(hunting): _negotiate 2024-03-29T10:45:59.500 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:59.511+0000 7fc1bbfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:45:59.500 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:59.511+0000 7fc1bbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:45:59.500 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:59.511+0000 7fc1bbfff640 10 monclient(hunting): my global_id is 4442 2024-03-29T10:45:59.500 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:59.511+0000 7fc1bbfff640 1 -- 172.21.15.88:0/3273302681 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc1b4004eb0 con 0x7fc1c80c9450 2024-03-29T10:45:59.501 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:59.511+0000 7fc1bbfff640 10 client.?.objecter ms_handle_connect 0x7fc1c80cc330 2024-03-29T10:45:59.501 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:59.511+0000 7fc1bbfff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:45:59.501 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:59.512+0000 7fc1bbfff640 1 -- 172.21.15.88:0/3273302681 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3925599974 0 0) 0x7fc1c81614e0 con 0x7fc1c80cc330 2024-03-29T10:45:59.501 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:59.512+0000 7fc1bbfff640 10 monclient(hunting): _negotiate 2024-03-29T10:45:59.501 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:59.512+0000 7fc1bbfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:45:59.501 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:59.512+0000 7fc1bbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:45:59.501 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:59.512+0000 7fc1bbfff640 10 monclient(hunting): my global_id is 4411 2024-03-29T10:45:59.501 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:59.512+0000 7fc1bbfff640 1 -- 172.21.15.88:0/3273302681 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc1c80c4270 con 0x7fc1c80cc330 2024-03-29T10:45:59.501 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:59.512+0000 7fc1bbfff640 1 -- 172.21.15.88:0/3273302681 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 364538965 0 0) 0x7fc1b4004eb0 con 0x7fc1c80c9450 2024-03-29T10:45:59.501 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:59.512+0000 7fc1bbfff640 1 -- 172.21.15.88:0/3273302681 --> v1:172.21.15.88:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc1c81614e0 con 0x7fc1c80c9450 2024-03-29T10:45:59.501 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:59.512+0000 7fc1bbfff640 1 -- 172.21.15.88:0/3273302681 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7fc1bc001710 con 0x7fc1c80c9450 2024-03-29T10:45:59.501 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:59.512+0000 7fc1bbfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:59.501 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:59.512+0000 7fc1bbfff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:45:59.501 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:59.512+0000 7fc1bbfff640 10 monclient(hunting): dump: 2024-03-29T10:45:59.502 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:45:59.502 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:45:59.502 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:45:59.502 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:45:59.502 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:59.502 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:45:59.502 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:45:59.502 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:45:59.502 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:45:59.502 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:45:59.502 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:59.512+0000 7fc1bbfff640 1 -- 172.21.15.88:0/3273302681 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1180811619 0 0) 0x7fc1c80c4270 con 0x7fc1c80cc330 2024-03-29T10:45:59.502 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:59.513+0000 7fc1bbfff640 1 -- 172.21.15.88:0/3273302681 --> v1:172.21.15.196:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc1b4004eb0 con 0x7fc1c80cc330 2024-03-29T10:45:59.502 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:59.513+0000 7fc1bbfff640 1 -- 172.21.15.88:0/3273302681 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7fc1c4001980 con 0x7fc1c80cc330 2024-03-29T10:45:59.503 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:59.513+0000 7fc1bbfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:59.503 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:59.513+0000 7fc1bbfff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T10:45:59.503 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:59.513+0000 7fc1bbfff640 10 monclient(hunting): dump: 2024-03-29T10:45:59.503 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:45:59.503 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:45:59.503 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:45:59.503 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:45:59.503 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:59.503 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:45:59.503 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:45:59.503 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:45:59.503 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:45:59.503 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:45:59.503 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:59.513+0000 7fc1bbfff640 1 -- 172.21.15.88:0/3273302681 <== mon.0 v1:172.21.15.88:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 116753506 0 0) 0x7fc1c80cb0a0 con 0x7fc1c8006da0 2024-03-29T10:45:59.504 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:59.513+0000 7fc1bbfff640 1 -- 172.21.15.88:0/3273302681 >> v1:172.21.15.88:6790/0 conn(0x7fc1c80c9450 legacy=0x7fc1c80c3450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:59.504 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:59.513+0000 7fc1bbfff640 1 -- 172.21.15.88:0/3273302681 >> v1:172.21.15.196:6789/0 conn(0x7fc1c80cc330 legacy=0x7fc1c80c3b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:59.504 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:59.513+0000 7fc1bbfff640 10 monclient: _finish_hunting 0 2024-03-29T10:45:59.504 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:59.513+0000 7fc1bbfff640 1 monclient: found mon.a 2024-03-29T10:45:59.504 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:59.513+0000 7fc1bbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:45:59.504 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:59.513+0000 7fc1bbfff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-03-29T10:45:59.504 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:59.513+0000 7fc1bbfff640 1 -- 172.21.15.88:0/3273302681 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1c80cbb10 con 0x7fc1c8006da0 2024-03-29T10:45:59.504 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:59.513+0000 7fc1bbfff640 10 monclient: _finish_auth 0 2024-03-29T10:45:59.504 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:59.513+0000 7fc1bbfff640 10 monclient: _check_auth_tickets 2024-03-29T10:45:59.504 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:59.513+0000 7fc1bbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:45:59.504 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:59.513+0000 7fc1d5fd6640 5 monclient: authenticate success, global_id 4419 2024-03-29T10:45:59.504 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:59.514+0000 7fc1d5fd6640 10 monclient: _renew_subs 2024-03-29T10:45:59.504 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:59.514+0000 7fc1d5fd6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-03-29T10:45:59.504 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:59.514+0000 7fc1d5fd6640 1 -- 172.21.15.88:0/3273302681 --> v1:172.21.15.88:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1c80cab80 con 0x7fc1c8006da0 2024-03-29T10:45:59.505 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:59.514+0000 7fc1d5fd6640 10 client.4419.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:45:59.505 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:59.514+0000 7fc1d5fd6640 10 monclient: _renew_subs 2024-03-29T10:45:59.505 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:59.514+0000 7fc1d5fd6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-03-29T10:45:59.505 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:59.514+0000 7fc1d5fd6640 1 -- 172.21.15.88:0/3273302681 --> v1:172.21.15.88:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1c802cab0 con 0x7fc1c8006da0 2024-03-29T10:45:59.505 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:59.514+0000 7fc1d5fd6640 1 librados: init done 2024-03-29T10:45:59.507 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:59.516+0000 7fc1bbfff640 1 -- 172.21.15.88:0/3273302681 <== mon.0 v1:172.21.15.88:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc1d004fdd0 con 0x7fc1c8006da0 2024-03-29T10:45:59.507 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:59.516+0000 7fc1d5fd6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:45:59.507 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:59.516+0000 7fc1bbfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:45:59.507 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:59.517+0000 7fc1bbfff640 1 -- 172.21.15.88:0/3273302681 <== mon.0 v1:172.21.15.88:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7fc1d004f730 con 0x7fc1c8006da0 2024-03-29T10:45:59.507 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:59.517+0000 7fc1bbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:59.507 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:59.517+0000 7fc1bbfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:45:59.507 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:59.517+0000 7fc1bbfff640 10 monclient: dump: 2024-03-29T10:45:59.507 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:45:59.507 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:45:59.508 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:45:59.508 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:45:59.508 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:59.508 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:45:59.508 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:45:59.508 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:45:59.508 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:45:59.508 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:45:59.508 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:59.517+0000 7fc1cd7fa640 4 set_mon_vals no callback set 2024-03-29T10:45:59.508 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:59.517+0000 7fc1d5fd6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:45:59.508 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:59.517+0000 7fc1d5fd6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-03-29T10:45:59.508 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:59.517+0000 7fc1d5fd6640 1 -- 172.21.15.88:0/3273302681 --> v1:172.21.15.88:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1c80cae80 con 0x7fc1c8006da0 2024-03-29T10:45:59.508 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:59.518+0000 7fc1bbfff640 1 -- 172.21.15.88:0/3273302681 <== mon.0 v1:172.21.15.88:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4235629220 0 0) 0x7fc1d0067ea0 con 0x7fc1c8006da0 2024-03-29T10:45:59.508 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:59.518+0000 7fc1bbfff640 1 -- 172.21.15.88:0/3273302681 <== mon.0 v1:172.21.15.88:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4257673582 0 0) 0x7fc1d00ff770 con 0x7fc1c8006da0 2024-03-29T10:45:59.508 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:59.518+0000 7fc1bbfff640 10 client.4419.objecter ms_dispatch 0x7fc1d0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:45:59.509 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:59.518+0000 7fc1bbfff640 3 client.4419.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:45:59.509 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:59.518+0000 7fc1bbfff640 3 client.4419.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:45:59.509 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:59.518+0000 7fc1bbfff640 20 client.4419.objecter dump_active .. 0 homeless 2024-03-29T10:45:59.509 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:59.519+0000 7fc1ceffd640 10 monclient: get_auth_request con 0x7fc1b406df30 auth_method 0 2024-03-29T10:45:59.509 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:59.519+0000 7fc1bbfff640 10 client.4419.objecter ms_handle_connect 0x7fc1b406df30 2024-03-29T10:45:59.512 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:59.527+0000 7fc1bbfff640 1 -- 172.21.15.88:0/3273302681 <== mon.0 v1:172.21.15.88:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc1d00cb130 con 0x7fc1c8006da0 2024-03-29T10:45:59.512 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:59.527+0000 7fc1bbfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:45:59.512 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:59.527+0000 7fc1bbfff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:45:59.833 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:59.850+0000 7fc1d5fd6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:45:59.833 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:59.850+0000 7fc1d5fd6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:45:59.833 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:59.850+0000 7fc1d5fd6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-03-29T10:45:59.833 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:59.850+0000 7fc1d5fd6640 1 -- 172.21.15.88:0/3273302681 --> v1:172.21.15.88:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc1c8005140 con 0x7fc1c8006da0 2024-03-29T10:45:59.833 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:59.850+0000 7fc1bbfff640 1 -- 172.21.15.88:0/3273302681 <== mon.0 v1:172.21.15.88:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 344506374) 0x7fc1d00cf620 con 0x7fc1c8006da0 2024-03-29T10:45:59.833 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:59.851+0000 7fc1bbfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:45:59.834 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:59.851+0000 7fc1bbfff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:45:59.835 INFO:teuthology.misc.health.smithi088.stdout: 2024-03-29T10:45:59.835 INFO:teuthology.misc.health.smithi088.stdout:{"epoch":10,"fsid":"869f5188-f92b-42dd-a46d-8c646e0388a9","created":"2024-03-29T10:40:40.285087+0000","modified":"2024-03-29T10:40:42.536230+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:40:42.536230+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a26c0c1c-8de0-438c-a3f6-3df10ea61574","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5de24915-8dec-49b6-8b16-f3170197c591","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ac054af3-d589-4565-9644-e6af75975c53","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3ec4f045-c714-4a22-bd1a-e07330b45ab3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"cddb9867-8e30-4951-9b13-3d1e78b9c45a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dc6d9911-785d-432f-8b53-56a1ccab302f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"77c88437-b587-496e-8b4d-1824ea8b5305","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ce963d96-5a8e-44c3-99de-c0dbebe234ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:45:59.838 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:59.855+0000 7fc1d5fd6640 10 librados: watch_flush enter 2024-03-29T10:45:59.839 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:59.855+0000 7fc1d5fd6640 10 librados: watch_flush exit 2024-03-29T10:45:59.839 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:59.855+0000 7fc1d5fd6640 20 client.4419.objecter shutdown clearing up homeless session... 2024-03-29T10:45:59.839 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:59.855+0000 7fc1d5fd6640 10 client.4419.objecter successfully canceled tick 2024-03-29T10:45:59.839 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:59.855+0000 7fc1d5fd6640 1 -- 172.21.15.88:0/3273302681 >> v1:172.21.15.88:6816/1752861349 conn(0x7fc1b406df30 legacy=0x7fc1b40703f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:59.839 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:59.855+0000 7fc1d5fd6640 10 monclient: shutdown 2024-03-29T10:45:59.839 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:59.855+0000 7fc1d5fd6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:45:59.839 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:59.855+0000 7fc1d5fd6640 1 -- 172.21.15.88:0/3273302681 >> v1:172.21.15.88:6789/0 conn(0x7fc1c8006da0 legacy=0x7fc1c8160dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:59.839 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:59.856+0000 7fc1d5fd6640 1 -- 172.21.15.88:0/3273302681 shutdown_connections 2024-03-29T10:45:59.839 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:59.856+0000 7fc1d5fd6640 1 -- 172.21.15.88:0/3273302681 >> 172.21.15.88:0/3273302681 conn(0x7fc1c8005c20 msgr2=0x7fc1c80392e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:45:59.839 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:59.856+0000 7fc1d5fd6640 1 -- 172.21.15.88:0/3273302681 shutdown_connections 2024-03-29T10:45:59.839 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:59.856+0000 7fc1d5fd6640 1 -- 172.21.15.88:0/3273302681 wait complete. 2024-03-29T10:45:59.840 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:45:59.858+0000 7fc1d5fd6640 1 librados: shutdown 2024-03-29T10:45:59.857 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:46:06.665 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:46:06.821 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:06.838+0000 7f7776138640 10 monclient: get_monmap_and_config 2024-03-29T10:46:06.821 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:06.838+0000 7f7776138640 10 monclient: build_initial_monmap 2024-03-29T10:46:06.821 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:06.838+0000 7f7776138640 10 monclient: monmap: 2024-03-29T10:46:06.822 INFO:teuthology.misc.health.smithi088.stderr:epoch 0 2024-03-29T10:46:06.822 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:46:06.822 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:46:06.839815+0000 2024-03-29T10:46:06.822 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:46:06.839815+0000 2024-03-29T10:46:06.822 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 0 (unknown) 2024-03-29T10:46:06.822 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:46:06.822 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-03-29T10:46:06.822 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-29T10:46:06.822 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-b 2024-03-29T10:46:06.822 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:46:06.823 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:06.839+0000 7f7776138640 1 Processor -- start 2024-03-29T10:46:06.823 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:06.840+0000 7f7776138640 1 -- start start 2024-03-29T10:46:06.823 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:06.840+0000 7f7776138640 10 monclient: init 2024-03-29T10:46:06.823 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:06.841+0000 7f7776138640 10 monclient: _reopen_session rank -1 2024-03-29T10:46:06.824 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:06.841+0000 7f7776138640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:46:06.824 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:06.841+0000 7f7776138640 10 monclient(hunting): picked mon.noname-c con 0x7f7768006a70 addr v1:172.21.15.196:6789/0 2024-03-29T10:46:06.824 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:06.841+0000 7f7776138640 10 monclient(hunting): picked mon.noname-a con 0x7f7768033060 addr v1:172.21.15.88:6789/0 2024-03-29T10:46:06.824 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:06.841+0000 7f7776138640 10 monclient(hunting): picked mon.noname-b con 0x7f77680c1e70 addr v1:172.21.15.88:6790/0 2024-03-29T10:46:06.824 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:06.841+0000 7f7776138640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f776802cfb0 con 0x7f7768033060 2024-03-29T10:46:06.824 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:06.841+0000 7f7776138640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f776802ae70 con 0x7f7768006a70 2024-03-29T10:46:06.824 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:06.841+0000 7f7776138640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77680c8110 con 0x7f77680c1e70 2024-03-29T10:46:06.824 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:06.841+0000 7f7776138640 10 monclient(hunting): _renew_subs 2024-03-29T10:46:06.824 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:06.841+0000 7f7776138640 10 monclient(hunting): authenticate will time out at 1089.077131s 2024-03-29T10:46:06.825 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:06.841+0000 7f7775136640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f7768006a70 0x7f77680c7a00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.88:49638/0 (socket says 172.21.15.88:49638) 2024-03-29T10:46:06.825 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:06.841+0000 7f7775136640 1 -- 172.21.15.88:0/3398371338 learned_addr learned my addr 172.21.15.88:0/3398371338 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-29T10:46:06.825 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:06.842+0000 7f7767fff640 1 -- 172.21.15.88:0/3398371338 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1674217269 0 0) 0x7f77680c8110 con 0x7f77680c1e70 2024-03-29T10:46:06.825 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:06.842+0000 7f7767fff640 10 monclient(hunting): _negotiate 2024-03-29T10:46:06.825 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:06.842+0000 7f7767fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:06.825 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:06.842+0000 7f7767fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:06.825 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:06.842+0000 7f7767fff640 10 monclient(hunting): my global_id is 4445 2024-03-29T10:46:06.825 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:06.842+0000 7f7767fff640 1 -- 172.21.15.88:0/3398371338 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f775c003520 con 0x7f77680c1e70 2024-03-29T10:46:06.825 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:06.842+0000 7f7767fff640 1 -- 172.21.15.88:0/3398371338 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 10628398 0 0) 0x7f775c003520 con 0x7f77680c1e70 2024-03-29T10:46:06.825 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:06.842+0000 7f7767fff640 1 -- 172.21.15.88:0/3398371338 >> v1:172.21.15.196:6789/0 conn(0x7f7768006a70 legacy=0x7f77680c7a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:06.826 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:06.842+0000 7f7767fff640 1 -- 172.21.15.88:0/3398371338 >> v1:172.21.15.88:6789/0 conn(0x7f7768033060 legacy=0x7f77680c1760 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:46:06.827 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:06.845+0000 7f7767fff640 10 monclient: _finish_hunting 0 2024-03-29T10:46:06.827 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:06.845+0000 7f7767fff640 1 monclient: found mon.noname-b 2024-03-29T10:46:06.828 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:06.845+0000 7f7767fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:46:06.828 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:06.845+0000 7f7767fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.88:6790/0 2024-03-29T10:46:06.828 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:06.845+0000 7f7767fff640 1 -- 172.21.15.88:0/3398371338 --> v1:172.21.15.88:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f776802d200 con 0x7f77680c1e70 2024-03-29T10:46:06.828 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:06.845+0000 7f7767fff640 10 monclient: _finish_auth 0 2024-03-29T10:46:06.828 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:06.845+0000 7f7767fff640 10 monclient: _check_auth_tickets 2024-03-29T10:46:06.828 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:06.845+0000 7f7767fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:46:06.828 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:06.845+0000 7f7767fff640 1 -- 172.21.15.88:0/3398371338 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f777004f730 con 0x7f77680c1e70 2024-03-29T10:46:06.828 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:06.845+0000 7f7767fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:06.828 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:06.845+0000 7f7767fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:46:06.828 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:06.845+0000 7f7767fff640 10 monclient: dump: 2024-03-29T10:46:06.828 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:46:06.828 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:46:06.828 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:46:06.828 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:46:06.829 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:06.829 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:46:06.829 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:46:06.829 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:46:06.829 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:46:06.829 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:46:06.829 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:06.845+0000 7f7776138640 5 monclient: authenticate success, global_id 4445 2024-03-29T10:46:06.829 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:06.845+0000 7f7776138640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:46:06.830 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:06.846+0000 7f7767fff640 1 -- 172.21.15.88:0/3398371338 <== mon.2 v1:172.21.15.88:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f77700008d0 con 0x7f77680c1e70 2024-03-29T10:46:06.830 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:06.846+0000 7f7767fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:46:06.830 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:06.846+0000 7f7776138640 10 monclient: get_monmap_and_config success 2024-03-29T10:46:06.830 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:06.846+0000 7f7767fff640 1 -- 172.21.15.88:0/3398371338 <== mon.2 v1:172.21.15.88:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f77700758b0 con 0x7f77680c1e70 2024-03-29T10:46:06.830 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:06.846+0000 7f7776138640 4 set_mon_vals no callback set 2024-03-29T10:46:06.830 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:06.846+0000 7f7767fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:06.830 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:06.846+0000 7f7767fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:46:06.830 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:06.846+0000 7f7767fff640 10 monclient: dump: 2024-03-29T10:46:06.831 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:46:06.831 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:46:06.831 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:46:06.831 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:46:06.831 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:06.831 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:46:06.831 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:46:06.831 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:46:06.831 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:46:06.831 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:46:06.831 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:06.847+0000 7f7776138640 10 monclient: shutdown 2024-03-29T10:46:06.831 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:06.847+0000 7f7776138640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:46:06.831 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:06.847+0000 7f7776138640 1 -- 172.21.15.88:0/3398371338 >> v1:172.21.15.88:6790/0 conn(0x7f77680c1e70 legacy=0x7f77680cfda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:06.831 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:06.847+0000 7f7776138640 1 -- 172.21.15.88:0/3398371338 shutdown_connections 2024-03-29T10:46:06.832 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:06.848+0000 7f7776138640 1 -- 172.21.15.88:0/3398371338 >> 172.21.15.88:0/3398371338 conn(0x7f77680058f0 msgr2=0x7f7768005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:46:06.832 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:06.849+0000 7f7776138640 1 -- 172.21.15.88:0/3398371338 shutdown_connections 2024-03-29T10:46:06.832 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:06.849+0000 7f7776138640 1 -- 172.21.15.88:0/3398371338 wait complete. 2024-03-29T10:46:06.833 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:06.850+0000 7f7776138640 10 monclient: build_initial_monmap 2024-03-29T10:46:06.834 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:06.850+0000 7f7776138640 10 monclient: monmap: 2024-03-29T10:46:06.834 INFO:teuthology.misc.health.smithi088.stderr:epoch 0 2024-03-29T10:46:06.834 INFO:teuthology.misc.health.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:46:06.834 INFO:teuthology.misc.health.smithi088.stderr:last_changed 0.000000 2024-03-29T10:46:06.834 INFO:teuthology.misc.health.smithi088.stderr:created 0.000000 2024-03-29T10:46:06.834 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 0 (unknown) 2024-03-29T10:46:06.834 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:46:06.834 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-03-29T10:46:06.834 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-29T10:46:06.834 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-c 2024-03-29T10:46:06.834 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:46:06.834 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:06.850+0000 7f7776138640 1 librados: starting msgr at 2024-03-29T10:46:06.834 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:06.851+0000 7f7776138640 1 librados: starting objecter 2024-03-29T10:46:06.834 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:06.851+0000 7f7776138640 1 Processor -- start 2024-03-29T10:46:06.835 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:06.851+0000 7f7776138640 1 -- start start 2024-03-29T10:46:06.835 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:06.851+0000 7f7776138640 1 librados: setting wanted keys 2024-03-29T10:46:06.835 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:06.851+0000 7f7776138640 1 librados: calling monclient init 2024-03-29T10:46:06.835 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:06.851+0000 7f7776138640 10 monclient: init 2024-03-29T10:46:06.840 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:06.852+0000 7f7776138640 10 monclient: _reopen_session rank -1 2024-03-29T10:46:06.840 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:06.852+0000 7f7776138640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:46:06.841 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:06.852+0000 7f7776138640 10 monclient(hunting): picked mon.noname-c con 0x7f7768006a70 addr v1:172.21.15.88:6790/0 2024-03-29T10:46:06.841 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:06.852+0000 7f7776138640 10 monclient(hunting): picked mon.noname-b con 0x7f77680c1e70 addr v1:172.21.15.196:6789/0 2024-03-29T10:46:06.841 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:06.852+0000 7f7776138640 10 monclient(hunting): picked mon.noname-a con 0x7f7768150cc0 addr v1:172.21.15.88:6789/0 2024-03-29T10:46:06.841 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:06.852+0000 7f7776138640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7768151150 con 0x7f7768150cc0 2024-03-29T10:46:06.841 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:06.852+0000 7f7776138640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77680c5290 con 0x7f77680c1e70 2024-03-29T10:46:06.841 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:06.852+0000 7f7776138640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77680c5420 con 0x7f7768006a70 2024-03-29T10:46:06.841 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:06.853+0000 7f7776138640 10 monclient(hunting): _renew_subs 2024-03-29T10:46:06.841 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:06.853+0000 7f7776138640 10 monclient(hunting): authenticate will time out at 1089.088963s 2024-03-29T10:46:06.841 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:06.853+0000 7f7774935640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f77680c1e70 0x7f77680c45c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.88:56550/0 (socket says 172.21.15.88:56550) 2024-03-29T10:46:06.841 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:06.853+0000 7f7774935640 1 -- 172.21.15.88:0/2741575390 learned_addr learned my addr 172.21.15.88:0/2741575390 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-29T10:46:06.841 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:06.853+0000 7f7765ffb640 10 client.?.objecter ms_handle_connect 0x7f77680c1e70 2024-03-29T10:46:06.841 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:06.853+0000 7f7765ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:46:06.842 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:06.854+0000 7f7765ffb640 1 -- 172.21.15.88:0/2741575390 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3266261912 0 0) 0x7f77680c5290 con 0x7f77680c1e70 2024-03-29T10:46:06.842 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:06.854+0000 7f7765ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:46:06.842 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:06.854+0000 7f7765ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:06.842 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:06.854+0000 7f7765ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:06.842 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:06.854+0000 7f7765ffb640 10 monclient(hunting): my global_id is 4417 2024-03-29T10:46:06.842 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:06.854+0000 7f7765ffb640 1 -- 172.21.15.88:0/2741575390 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7754004eb0 con 0x7f77680c1e70 2024-03-29T10:46:06.842 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:06.854+0000 7f7765ffb640 1 -- 172.21.15.88:0/2741575390 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 902077734 0 0) 0x7f7754004eb0 con 0x7f77680c1e70 2024-03-29T10:46:06.842 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:06.855+0000 7f7765ffb640 1 -- 172.21.15.88:0/2741575390 --> v1:172.21.15.196:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f77680c5290 con 0x7f77680c1e70 2024-03-29T10:46:06.842 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:06.855+0000 7f7765ffb640 1 -- 172.21.15.88:0/2741575390 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f776c001960 con 0x7f77680c1e70 2024-03-29T10:46:06.842 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:06.855+0000 7f7765ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:06.842 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:06.855+0000 7f7765ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:46:06.842 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:06.855+0000 7f7765ffb640 10 monclient(hunting): dump: 2024-03-29T10:46:06.843 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:46:06.843 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:46:06.843 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:46:06.843 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:46:06.843 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:06.843 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:46:06.843 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:46:06.843 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:46:06.843 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:46:06.843 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:46:06.843 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:06.855+0000 7f7765ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:46:06.843 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:06.855+0000 7f7765ffb640 1 -- 172.21.15.88:0/2741575390 <== mon.1 v1:172.21.15.196:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3952518820 0 0) 0x7f77680c5290 con 0x7f77680c1e70 2024-03-29T10:46:06.843 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:06.855+0000 7f7765ffb640 1 -- 172.21.15.88:0/2741575390 >> v1:172.21.15.88:6790/0 conn(0x7f7768006a70 legacy=0x7f776815cce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:06.843 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:06.856+0000 7f7765ffb640 1 -- 172.21.15.88:0/2741575390 >> v1:172.21.15.88:6789/0 conn(0x7f7768150cc0 legacy=0x7f77680bad80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:06.844 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:06.856+0000 7f7765ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:46:06.844 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:06.856+0000 7f7765ffb640 1 monclient: found mon.b 2024-03-29T10:46:06.844 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:06.856+0000 7f7765ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:46:06.844 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:06.856+0000 7f7765ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T10:46:06.844 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:06.856+0000 7f7765ffb640 1 -- 172.21.15.88:0/2741575390 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f776814d330 con 0x7f77680c1e70 2024-03-29T10:46:06.844 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:06.856+0000 7f7765ffb640 10 monclient: _finish_auth 0 2024-03-29T10:46:06.844 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:06.856+0000 7f7765ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:46:06.844 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:06.856+0000 7f7765ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:46:06.844 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:06.856+0000 7f7765ffb640 10 client.?.objecter ms_handle_connect 0x7f7768006a70 2024-03-29T10:46:06.844 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:06.856+0000 7f7765ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:46:06.844 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:06.856+0000 7f7776138640 5 monclient: authenticate success, global_id 4417 2024-03-29T10:46:06.844 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:06.856+0000 7f7776138640 10 monclient: _renew_subs 2024-03-29T10:46:06.844 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:06.856+0000 7f7776138640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T10:46:06.844 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:06.856+0000 7f7776138640 1 -- 172.21.15.88:0/2741575390 --> v1:172.21.15.196:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7768032020 con 0x7f77680c1e70 2024-03-29T10:46:06.845 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:06.856+0000 7f7776138640 10 client.4417.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:46:06.845 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:06.856+0000 7f7776138640 10 monclient: _renew_subs 2024-03-29T10:46:06.845 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:06.856+0000 7f7776138640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T10:46:06.845 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:06.856+0000 7f7776138640 1 -- 172.21.15.88:0/2741575390 --> v1:172.21.15.196:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f77680bb670 con 0x7f77680c1e70 2024-03-29T10:46:06.845 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:06.856+0000 7f7776138640 1 librados: init done 2024-03-29T10:46:06.845 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:06.856+0000 7f7765ffb640 1 -- 172.21.15.88:0/2741575390 <== mon.1 v1:172.21.15.196:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f776c001ac0 con 0x7f77680c1e70 2024-03-29T10:46:06.845 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:06.856+0000 7f7765ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:46:06.845 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:06.856+0000 7f7765ffb640 1 -- 172.21.15.88:0/2741575390 <== mon.1 v1:172.21.15.196:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f776c003a40 con 0x7f77680c1e70 2024-03-29T10:46:06.845 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:06.857+0000 7f7765ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:06.845 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:06.857+0000 7f7765ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:46:06.845 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:06.857+0000 7f7765ffb640 10 monclient: dump: 2024-03-29T10:46:06.845 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:46:06.845 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:46:06.845 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:46:06.845 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:46:06.846 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:06.846 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:46:06.846 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:46:06.846 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:46:06.846 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:46:06.846 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:46:06.846 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:06.857+0000 7f77677fe640 4 set_mon_vals no callback set 2024-03-29T10:46:06.846 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:06.858+0000 7f7776138640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:46:06.846 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:06.858+0000 7f7776138640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:46:06.846 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:06.858+0000 7f7776138640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T10:46:06.846 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:06.858+0000 7f7776138640 1 -- 172.21.15.88:0/2741575390 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7750005080 con 0x7f77680c1e70 2024-03-29T10:46:06.847 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:06.859+0000 7f7765ffb640 1 -- 172.21.15.88:0/2741575390 <== mon.1 v1:172.21.15.196:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4235629220 0 0) 0x7f776c003ba0 con 0x7f77680c1e70 2024-03-29T10:46:06.847 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:06.860+0000 7f7765ffb640 1 -- 172.21.15.88:0/2741575390 <== mon.1 v1:172.21.15.196:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4257673582 0 0) 0x7f776c084b10 con 0x7f77680c1e70 2024-03-29T10:46:06.847 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:06.860+0000 7f7765ffb640 10 client.4417.objecter ms_dispatch 0x7f7770000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:46:06.848 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:06.860+0000 7f7765ffb640 3 client.4417.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:46:06.848 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:06.860+0000 7f7765ffb640 3 client.4417.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:46:06.848 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:06.860+0000 7f7765ffb640 20 client.4417.objecter dump_active .. 0 homeless 2024-03-29T10:46:06.848 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:06.861+0000 7f7775136640 10 monclient: get_auth_request con 0x7f775406dc60 auth_method 0 2024-03-29T10:46:06.848 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:06.862+0000 7f7765ffb640 10 client.4417.objecter ms_handle_connect 0x7f775406dc60 2024-03-29T10:46:06.849 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:06.866+0000 7f7765ffb640 1 -- 172.21.15.88:0/2741575390 <== mon.1 v1:172.21.15.196:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f776c0503f0 con 0x7f77680c1e70 2024-03-29T10:46:06.849 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:06.867+0000 7f7765ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:46:06.849 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:06.867+0000 7f7765ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:46:07.198 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:07.216+0000 7f7776138640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:46:07.198 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:07.216+0000 7f7776138640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:46:07.199 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:07.216+0000 7f7776138640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T10:46:07.199 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:07.216+0000 7f7776138640 1 -- 172.21.15.88:0/2741575390 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7750005330 con 0x7f77680c1e70 2024-03-29T10:46:07.199 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:07.217+0000 7f7765ffb640 1 -- 172.21.15.88:0/2741575390 <== mon.1 v1:172.21.15.196:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 344506374) 0x7f776c052320 con 0x7f77680c1e70 2024-03-29T10:46:07.199 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:07.217+0000 7f7765ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:46:07.199 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:07.217+0000 7f7765ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:46:07.200 INFO:teuthology.misc.health.smithi088.stdout: 2024-03-29T10:46:07.200 INFO:teuthology.misc.health.smithi088.stdout:{"epoch":10,"fsid":"869f5188-f92b-42dd-a46d-8c646e0388a9","created":"2024-03-29T10:40:40.285087+0000","modified":"2024-03-29T10:40:42.536230+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:40:42.536230+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a26c0c1c-8de0-438c-a3f6-3df10ea61574","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5de24915-8dec-49b6-8b16-f3170197c591","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ac054af3-d589-4565-9644-e6af75975c53","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3ec4f045-c714-4a22-bd1a-e07330b45ab3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"cddb9867-8e30-4951-9b13-3d1e78b9c45a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dc6d9911-785d-432f-8b53-56a1ccab302f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"77c88437-b587-496e-8b4d-1824ea8b5305","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ce963d96-5a8e-44c3-99de-c0dbebe234ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:46:07.203 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:07.220+0000 7f7776138640 10 librados: watch_flush enter 2024-03-29T10:46:07.203 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:07.220+0000 7f7776138640 10 librados: watch_flush exit 2024-03-29T10:46:07.203 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:07.220+0000 7f7776138640 20 client.4417.objecter shutdown clearing up homeless session... 2024-03-29T10:46:07.203 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:07.220+0000 7f7776138640 10 client.4417.objecter successfully canceled tick 2024-03-29T10:46:07.204 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:07.220+0000 7f7776138640 1 -- 172.21.15.88:0/2741575390 >> v1:172.21.15.88:6816/1752861349 conn(0x7f775406dc60 legacy=0x7f7754070120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:07.204 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:07.220+0000 7f7776138640 10 monclient: shutdown 2024-03-29T10:46:07.204 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:07.220+0000 7f7776138640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:46:07.204 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:07.220+0000 7f7776138640 1 -- 172.21.15.88:0/2741575390 >> v1:172.21.15.196:6789/0 conn(0x7f77680c1e70 legacy=0x7f77680c45c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:07.204 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:07.221+0000 7f7776138640 1 -- 172.21.15.88:0/2741575390 shutdown_connections 2024-03-29T10:46:07.204 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:07.221+0000 7f7776138640 1 -- 172.21.15.88:0/2741575390 >> 172.21.15.88:0/2741575390 conn(0x7f77680058f0 msgr2=0x7f77680bc530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:46:07.204 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:07.221+0000 7f7776138640 1 -- 172.21.15.88:0/2741575390 shutdown_connections 2024-03-29T10:46:07.204 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:07.221+0000 7f7776138640 1 -- 172.21.15.88:0/2741575390 wait complete. 2024-03-29T10:46:07.204 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:07.221+0000 7f7776138640 1 librados: shutdown 2024-03-29T10:46:07.218 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:46:14.025 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:46:14.169 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:14.186+0000 7f9de46ef640 10 monclient: get_monmap_and_config 2024-03-29T10:46:14.169 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:14.186+0000 7f9de46ef640 10 monclient: build_initial_monmap 2024-03-29T10:46:14.169 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:14.186+0000 7f9de46ef640 10 monclient: monmap: 2024-03-29T10:46:14.169 INFO:teuthology.misc.health.smithi088.stderr:epoch 0 2024-03-29T10:46:14.170 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:46:14.170 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:46:14.187730+0000 2024-03-29T10:46:14.170 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:46:14.187730+0000 2024-03-29T10:46:14.170 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 0 (unknown) 2024-03-29T10:46:14.170 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:46:14.170 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-03-29T10:46:14.170 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-29T10:46:14.170 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-b 2024-03-29T10:46:14.170 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:46:14.171 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:14.187+0000 7f9de46ef640 1 Processor -- start 2024-03-29T10:46:14.171 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:14.188+0000 7f9de46ef640 1 -- start start 2024-03-29T10:46:14.171 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:14.188+0000 7f9de46ef640 10 monclient: init 2024-03-29T10:46:14.172 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:14.189+0000 7f9de46ef640 10 monclient: _reopen_session rank -1 2024-03-29T10:46:14.172 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:14.189+0000 7f9de46ef640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:46:14.172 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:14.189+0000 7f9de46ef640 10 monclient(hunting): picked mon.noname-b con 0x7f9ddc1422b0 addr v1:172.21.15.88:6790/0 2024-03-29T10:46:14.172 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:14.189+0000 7f9de46ef640 10 monclient(hunting): picked mon.noname-a con 0x7f9ddc142690 addr v1:172.21.15.88:6789/0 2024-03-29T10:46:14.172 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:14.189+0000 7f9de46ef640 10 monclient(hunting): picked mon.noname-c con 0x7f9ddc0b17d0 addr v1:172.21.15.196:6789/0 2024-03-29T10:46:14.172 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:14.189+0000 7f9de46ef640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ddc142c80 con 0x7f9ddc142690 2024-03-29T10:46:14.172 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:14.189+0000 7f9de46ef640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ddc0aae40 con 0x7f9ddc0b17d0 2024-03-29T10:46:14.172 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:14.189+0000 7f9de46ef640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ddc13fbd0 con 0x7f9ddc1422b0 2024-03-29T10:46:14.172 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:14.189+0000 7f9de46ef640 10 monclient(hunting): _renew_subs 2024-03-29T10:46:14.173 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:14.189+0000 7f9de46ef640 10 monclient(hunting): authenticate will time out at 1096.425717s 2024-03-29T10:46:14.173 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:14.189+0000 7f9de2c8c640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f9ddc0b17d0 0x7f9ddc0b2ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.88:50672/0 (socket says 172.21.15.88:50672) 2024-03-29T10:46:14.173 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:14.189+0000 7f9de2c8c640 1 -- 172.21.15.88:0/2330271376 learned_addr learned my addr 172.21.15.88:0/2330271376 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-29T10:46:14.174 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:14.190+0000 7f9de1489640 1 -- 172.21.15.88:0/2330271376 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1024004070 0 0) 0x7f9ddc0aae40 con 0x7f9ddc0b17d0 2024-03-29T10:46:14.174 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:14.190+0000 7f9de1489640 10 monclient(hunting): _negotiate 2024-03-29T10:46:14.174 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:14.190+0000 7f9de1489640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:14.174 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:14.190+0000 7f9de1489640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:14.174 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:14.190+0000 7f9de1489640 10 monclient(hunting): my global_id is 4420 2024-03-29T10:46:14.174 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:14.190+0000 7f9de1489640 1 -- 172.21.15.88:0/2330271376 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9dc8003520 con 0x7f9ddc0b17d0 2024-03-29T10:46:14.174 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:14.190+0000 7f9de1489640 1 -- 172.21.15.88:0/2330271376 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3246451862 0 0) 0x7f9ddc142c80 con 0x7f9ddc142690 2024-03-29T10:46:14.174 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:14.190+0000 7f9de1489640 10 monclient(hunting): _negotiate 2024-03-29T10:46:14.174 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:14.190+0000 7f9de1489640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:14.174 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:14.190+0000 7f9de1489640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:14.175 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:14.190+0000 7f9de1489640 10 monclient(hunting): my global_id is 4422 2024-03-29T10:46:14.175 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:14.190+0000 7f9de1489640 1 -- 172.21.15.88:0/2330271376 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9ddc0aae40 con 0x7f9ddc142690 2024-03-29T10:46:14.175 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:14.190+0000 7f9de1489640 1 -- 172.21.15.88:0/2330271376 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2480162264 0 0) 0x7f9dc8003520 con 0x7f9ddc0b17d0 2024-03-29T10:46:14.175 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:14.190+0000 7f9de1489640 1 -- 172.21.15.88:0/2330271376 >> v1:172.21.15.88:6790/0 conn(0x7f9ddc1422b0 legacy=0x7f9ddc13f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:14.175 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:14.190+0000 7f9de1489640 1 -- 172.21.15.88:0/2330271376 >> v1:172.21.15.88:6789/0 conn(0x7f9ddc142690 legacy=0x7f9ddc0b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:14.175 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:14.190+0000 7f9de1489640 10 monclient: _finish_hunting 0 2024-03-29T10:46:14.175 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:14.190+0000 7f9de1489640 1 monclient: found mon.noname-c 2024-03-29T10:46:14.175 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:14.190+0000 7f9de1489640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:46:14.175 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:14.190+0000 7f9de1489640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.196:6789/0 2024-03-29T10:46:14.175 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:14.190+0000 7f9de1489640 1 -- 172.21.15.88:0/2330271376 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ddc151940 con 0x7f9ddc0b17d0 2024-03-29T10:46:14.175 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:14.190+0000 7f9de1489640 10 monclient: _finish_auth 0 2024-03-29T10:46:14.175 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:14.190+0000 7f9de1489640 10 monclient: _check_auth_tickets 2024-03-29T10:46:14.175 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:14.190+0000 7f9de1489640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:46:14.175 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:14.190+0000 7f9de1489640 1 -- 172.21.15.88:0/2330271376 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f9dd4001a10 con 0x7f9ddc0b17d0 2024-03-29T10:46:14.176 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:14.190+0000 7f9de1489640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:14.176 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:14.190+0000 7f9de1489640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:46:14.176 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:14.190+0000 7f9de1489640 10 monclient: dump: 2024-03-29T10:46:14.176 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:46:14.176 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:46:14.176 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:46:14.176 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:46:14.176 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:14.176 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:46:14.176 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:46:14.176 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:46:14.176 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:46:14.176 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:46:14.177 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:14.190+0000 7f9de46ef640 5 monclient: authenticate success, global_id 4420 2024-03-29T10:46:14.177 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:14.190+0000 7f9de46ef640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:46:14.177 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:14.191+0000 7f9de1489640 1 -- 172.21.15.88:0/2330271376 <== mon.1 v1:172.21.15.196:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9dd4001b70 con 0x7f9ddc0b17d0 2024-03-29T10:46:14.177 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:14.191+0000 7f9de1489640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:46:14.177 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:14.191+0000 7f9de1489640 1 -- 172.21.15.88:0/2330271376 <== mon.1 v1:172.21.15.196:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f9dd4001f40 con 0x7f9ddc0b17d0 2024-03-29T10:46:14.177 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:14.191+0000 7f9de1489640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:14.177 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:14.191+0000 7f9de1489640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:46:14.177 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:14.191+0000 7f9de1489640 10 monclient: dump: 2024-03-29T10:46:14.177 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:46:14.177 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:46:14.177 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:46:14.177 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:46:14.177 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:14.177 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:46:14.178 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:46:14.178 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:46:14.178 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:46:14.178 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:46:14.178 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:14.191+0000 7f9de46ef640 10 monclient: get_monmap_and_config success 2024-03-29T10:46:14.178 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:14.191+0000 7f9de46ef640 4 set_mon_vals no callback set 2024-03-29T10:46:14.182 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:14.192+0000 7f9de46ef640 10 monclient: shutdown 2024-03-29T10:46:14.182 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:14.192+0000 7f9de46ef640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:46:14.182 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:14.192+0000 7f9de46ef640 1 -- 172.21.15.88:0/2330271376 >> v1:172.21.15.196:6789/0 conn(0x7f9ddc0b17d0 legacy=0x7f9ddc0b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:14.182 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:14.192+0000 7f9de46ef640 1 -- 172.21.15.88:0/2330271376 shutdown_connections 2024-03-29T10:46:14.182 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:14.192+0000 7f9de46ef640 1 -- 172.21.15.88:0/2330271376 >> 172.21.15.88:0/2330271376 conn(0x7f9ddc066a00 msgr2=0x7f9ddc066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:46:14.182 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:14.193+0000 7f9de46ef640 1 -- 172.21.15.88:0/2330271376 shutdown_connections 2024-03-29T10:46:14.182 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:14.193+0000 7f9de46ef640 1 -- 172.21.15.88:0/2330271376 wait complete. 2024-03-29T10:46:14.182 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:14.194+0000 7f9de46ef640 10 monclient: build_initial_monmap 2024-03-29T10:46:14.182 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:14.194+0000 7f9de46ef640 10 monclient: monmap: 2024-03-29T10:46:14.182 INFO:teuthology.misc.health.smithi088.stderr:epoch 0 2024-03-29T10:46:14.183 INFO:teuthology.misc.health.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:46:14.183 INFO:teuthology.misc.health.smithi088.stderr:last_changed 0.000000 2024-03-29T10:46:14.183 INFO:teuthology.misc.health.smithi088.stderr:created 0.000000 2024-03-29T10:46:14.183 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 0 (unknown) 2024-03-29T10:46:14.183 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:46:14.183 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-03-29T10:46:14.183 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-29T10:46:14.183 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-c 2024-03-29T10:46:14.183 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:46:14.183 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:14.194+0000 7f9de46ef640 1 librados: starting msgr at 2024-03-29T10:46:14.183 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:14.194+0000 7f9de46ef640 1 librados: starting objecter 2024-03-29T10:46:14.183 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:14.194+0000 7f9de46ef640 1 Processor -- start 2024-03-29T10:46:14.183 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:14.195+0000 7f9de46ef640 1 -- start start 2024-03-29T10:46:14.183 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:14.195+0000 7f9de46ef640 1 librados: setting wanted keys 2024-03-29T10:46:14.184 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:14.195+0000 7f9de46ef640 1 librados: calling monclient init 2024-03-29T10:46:14.184 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:14.195+0000 7f9de46ef640 10 monclient: init 2024-03-29T10:46:14.184 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:14.195+0000 7f9de46ef640 10 monclient: _reopen_session rank -1 2024-03-29T10:46:14.184 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:14.196+0000 7f9de46ef640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:46:14.184 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:14.196+0000 7f9de46ef640 10 monclient(hunting): picked mon.noname-b con 0x7f9ddc0b17d0 addr v1:172.21.15.196:6789/0 2024-03-29T10:46:14.184 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:14.196+0000 7f9de46ef640 10 monclient(hunting): picked mon.noname-a con 0x7f9ddc1422b0 addr v1:172.21.15.88:6789/0 2024-03-29T10:46:14.184 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:14.196+0000 7f9de46ef640 10 monclient(hunting): picked mon.noname-c con 0x7f9ddc142690 addr v1:172.21.15.88:6790/0 2024-03-29T10:46:14.184 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:14.196+0000 7f9de46ef640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ddc14e2b0 con 0x7f9ddc1422b0 2024-03-29T10:46:14.184 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:14.196+0000 7f9de46ef640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ddc0ac7a0 con 0x7f9ddc0b17d0 2024-03-29T10:46:14.184 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:14.196+0000 7f9de46ef640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ddc0aca80 con 0x7f9ddc142690 2024-03-29T10:46:14.184 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:14.196+0000 7f9de46ef640 10 monclient(hunting): _renew_subs 2024-03-29T10:46:14.184 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:14.196+0000 7f9de46ef640 10 monclient(hunting): authenticate will time out at 1096.432266s 2024-03-29T10:46:14.184 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:14.196+0000 7f9de1c8a640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f9ddc1422b0 0x7f9ddc14d490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:35454/0 (socket says 172.21.15.88:35454) 2024-03-29T10:46:14.185 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:14.196+0000 7f9de1c8a640 1 -- 172.21.15.88:0/1235031096 learned_addr learned my addr 172.21.15.88:0/1235031096 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-29T10:46:14.185 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:14.196+0000 7f9dd2ffd640 10 client.?.objecter ms_handle_connect 0x7f9ddc142690 2024-03-29T10:46:14.185 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:14.196+0000 7f9dd2ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:46:14.185 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:14.196+0000 7f9dd2ffd640 10 client.?.objecter ms_handle_connect 0x7f9ddc1422b0 2024-03-29T10:46:14.185 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:14.196+0000 7f9dd2ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:46:14.185 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:14.196+0000 7f9dd2ffd640 10 client.?.objecter ms_handle_connect 0x7f9ddc0b17d0 2024-03-29T10:46:14.185 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:14.196+0000 7f9dd2ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:46:14.185 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:14.197+0000 7f9dd2ffd640 1 -- 172.21.15.88:0/1235031096 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1640353773 0 0) 0x7f9ddc0ac7a0 con 0x7f9ddc0b17d0 2024-03-29T10:46:14.185 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:14.197+0000 7f9dd2ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:46:14.185 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:14.197+0000 7f9dd2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:14.185 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:14.197+0000 7f9dd2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:14.185 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:14.197+0000 7f9dd2ffd640 10 monclient(hunting): my global_id is 4423 2024-03-29T10:46:14.185 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:14.197+0000 7f9dd2ffd640 1 -- 172.21.15.88:0/1235031096 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9dc0004eb0 con 0x7f9ddc0b17d0 2024-03-29T10:46:14.186 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:14.197+0000 7f9dd2ffd640 1 -- 172.21.15.88:0/1235031096 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4268775855 0 0) 0x7f9ddc0aca80 con 0x7f9ddc142690 2024-03-29T10:46:14.186 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:14.197+0000 7f9dd2ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:46:14.186 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:14.197+0000 7f9dd2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:14.186 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:14.197+0000 7f9dd2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:14.186 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:14.197+0000 7f9dd2ffd640 10 monclient(hunting): my global_id is 4451 2024-03-29T10:46:14.186 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:14.197+0000 7f9dd2ffd640 1 -- 172.21.15.88:0/1235031096 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9dc00051d0 con 0x7f9ddc142690 2024-03-29T10:46:14.186 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:14.198+0000 7f9dd2ffd640 1 -- 172.21.15.88:0/1235031096 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2004878938 0 0) 0x7f9dc00051d0 con 0x7f9ddc142690 2024-03-29T10:46:14.186 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:14.198+0000 7f9dd2ffd640 1 -- 172.21.15.88:0/1235031096 --> v1:172.21.15.88:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9ddc0aca80 con 0x7f9ddc142690 2024-03-29T10:46:14.186 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:14.198+0000 7f9dd2ffd640 1 -- 172.21.15.88:0/1235031096 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f9dd4001e80 con 0x7f9ddc142690 2024-03-29T10:46:14.186 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:14.198+0000 7f9dd2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:14.186 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:14.198+0000 7f9dd2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:46:14.186 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:14.198+0000 7f9dd2ffd640 10 monclient(hunting): dump: 2024-03-29T10:46:14.186 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:46:14.187 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:46:14.187 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:46:14.187 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:46:14.187 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:14.187 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:46:14.187 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:46:14.187 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:46:14.187 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:46:14.187 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:46:14.187 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:14.198+0000 7f9dd2ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:46:14.187 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:14.198+0000 7f9dd2ffd640 1 -- 172.21.15.88:0/1235031096 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2355805015 0 0) 0x7f9dc0004eb0 con 0x7f9ddc0b17d0 2024-03-29T10:46:14.187 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:14.198+0000 7f9dd2ffd640 1 -- 172.21.15.88:0/1235031096 --> v1:172.21.15.196:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9dc00051d0 con 0x7f9ddc0b17d0 2024-03-29T10:46:14.188 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:14.198+0000 7f9dd2ffd640 1 -- 172.21.15.88:0/1235031096 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f9dd8001930 con 0x7f9ddc0b17d0 2024-03-29T10:46:14.188 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:14.198+0000 7f9dd2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:14.188 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:14.198+0000 7f9dd2ffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T10:46:14.188 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:14.198+0000 7f9dd2ffd640 10 monclient(hunting): dump: 2024-03-29T10:46:14.188 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:46:14.188 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:46:14.188 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:46:14.188 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:46:14.188 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:14.188 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:46:14.188 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:46:14.188 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:46:14.188 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:46:14.188 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:46:14.188 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:14.198+0000 7f9dd2ffd640 1 -- 172.21.15.88:0/1235031096 <== mon.1 v1:172.21.15.196:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2631152792 0 0) 0x7f9dc00051d0 con 0x7f9ddc0b17d0 2024-03-29T10:46:14.189 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:14.198+0000 7f9dd2ffd640 1 -- 172.21.15.88:0/1235031096 >> v1:172.21.15.88:6790/0 conn(0x7f9ddc142690 legacy=0x7f9ddc14dba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:14.189 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:14.198+0000 7f9dd2ffd640 1 -- 172.21.15.88:0/1235031096 >> v1:172.21.15.88:6789/0 conn(0x7f9ddc1422b0 legacy=0x7f9ddc14d490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:14.189 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:14.198+0000 7f9dd2ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:46:14.189 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:14.198+0000 7f9dd2ffd640 1 monclient: found mon.b 2024-03-29T10:46:14.189 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:14.198+0000 7f9dd2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:46:14.189 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:14.198+0000 7f9dd2ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T10:46:14.189 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:14.198+0000 7f9dd2ffd640 1 -- 172.21.15.88:0/1235031096 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ddc0ab540 con 0x7f9ddc0b17d0 2024-03-29T10:46:14.189 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:14.198+0000 7f9dd2ffd640 10 monclient: _finish_auth 0 2024-03-29T10:46:14.189 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:14.198+0000 7f9dd2ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:46:14.189 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:14.198+0000 7f9dd2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:46:14.189 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:14.199+0000 7f9de46ef640 5 monclient: authenticate success, global_id 4423 2024-03-29T10:46:14.189 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:14.199+0000 7f9de46ef640 10 monclient: _renew_subs 2024-03-29T10:46:14.189 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:14.199+0000 7f9de46ef640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T10:46:14.190 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:14.199+0000 7f9de46ef640 1 -- 172.21.15.88:0/1235031096 --> v1:172.21.15.196:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ddc0aba60 con 0x7f9ddc0b17d0 2024-03-29T10:46:14.190 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:14.199+0000 7f9de46ef640 10 client.4423.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:46:14.190 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:14.199+0000 7f9de46ef640 10 monclient: _renew_subs 2024-03-29T10:46:14.190 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:14.199+0000 7f9de46ef640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T10:46:14.190 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:14.199+0000 7f9de46ef640 1 -- 172.21.15.88:0/1235031096 --> v1:172.21.15.196:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ddc12eb50 con 0x7f9ddc0b17d0 2024-03-29T10:46:14.190 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:14.199+0000 7f9de46ef640 1 librados: init done 2024-03-29T10:46:14.191 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:14.200+0000 7f9dd2ffd640 1 -- 172.21.15.88:0/1235031096 <== mon.1 v1:172.21.15.196:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9dd8003610 con 0x7f9ddc0b17d0 2024-03-29T10:46:14.191 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:14.200+0000 7f9dd2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:46:14.191 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:14.200+0000 7f9de0c88640 4 set_mon_vals no callback set 2024-03-29T10:46:14.191 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:14.200+0000 7f9dd2ffd640 1 -- 172.21.15.88:0/1235031096 <== mon.1 v1:172.21.15.196:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f9dd80039a0 con 0x7f9ddc0b17d0 2024-03-29T10:46:14.192 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:14.200+0000 7f9dd2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:14.192 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:14.200+0000 7f9dd2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:46:14.192 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:14.200+0000 7f9dd2ffd640 10 monclient: dump: 2024-03-29T10:46:14.192 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:46:14.192 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:46:14.192 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:46:14.192 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:46:14.192 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:14.192 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:46:14.192 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:46:14.192 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:46:14.192 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:46:14.192 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:46:14.193 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:14.200+0000 7f9de46ef640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:46:14.193 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:14.201+0000 7f9de46ef640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:46:14.193 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:14.201+0000 7f9de46ef640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T10:46:14.193 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:14.201+0000 7f9de46ef640 1 -- 172.21.15.88:0/1235031096 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9dc4005080 con 0x7f9ddc0b17d0 2024-03-29T10:46:14.193 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:14.203+0000 7f9dd2ffd640 1 -- 172.21.15.88:0/1235031096 <== mon.1 v1:172.21.15.196:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4235629220 0 0) 0x7f9dd8003b00 con 0x7f9ddc0b17d0 2024-03-29T10:46:14.193 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:14.204+0000 7f9de1c8a640 10 monclient: get_auth_request con 0x7f9dc006e250 auth_method 0 2024-03-29T10:46:14.193 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:14.204+0000 7f9dd2ffd640 1 -- 172.21.15.88:0/1235031096 <== mon.1 v1:172.21.15.196:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4257673582 0 0) 0x7f9dd8053440 con 0x7f9ddc0b17d0 2024-03-29T10:46:14.193 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:14.204+0000 7f9dd2ffd640 10 client.4423.objecter ms_dispatch 0x7f9ddc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:46:14.193 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:14.204+0000 7f9dd2ffd640 3 client.4423.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:46:14.193 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:14.204+0000 7f9dd2ffd640 3 client.4423.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:46:14.193 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:14.204+0000 7f9dd2ffd640 20 client.4423.objecter dump_active .. 0 homeless 2024-03-29T10:46:14.193 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:14.204+0000 7f9dd2ffd640 10 client.4423.objecter ms_handle_connect 0x7f9dc006e250 2024-03-29T10:46:14.195 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:14.212+0000 7f9dd2ffd640 1 -- 172.21.15.88:0/1235031096 <== mon.1 v1:172.21.15.196:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9dd804f6e0 con 0x7f9ddc0b17d0 2024-03-29T10:46:14.195 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:14.212+0000 7f9dd2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:46:14.195 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:14.212+0000 7f9dd2ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:46:14.608 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:14.626+0000 7f9de46ef640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:46:14.609 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:14.626+0000 7f9de46ef640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:46:14.609 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:14.626+0000 7f9de46ef640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T10:46:14.609 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:14.626+0000 7f9de46ef640 1 -- 172.21.15.88:0/1235031096 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9dc4005330 con 0x7f9ddc0b17d0 2024-03-29T10:46:14.609 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:14.627+0000 7f9dd2ffd640 1 -- 172.21.15.88:0/1235031096 <== mon.1 v1:172.21.15.196:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 344506374) 0x7f9dd804f880 con 0x7f9ddc0b17d0 2024-03-29T10:46:14.609 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:14.627+0000 7f9dd2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:46:14.610 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:14.627+0000 7f9dd2ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:46:14.610 INFO:teuthology.misc.health.smithi088.stdout: 2024-03-29T10:46:14.610 INFO:teuthology.misc.health.smithi088.stdout:{"epoch":10,"fsid":"869f5188-f92b-42dd-a46d-8c646e0388a9","created":"2024-03-29T10:40:40.285087+0000","modified":"2024-03-29T10:40:42.536230+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:40:42.536230+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a26c0c1c-8de0-438c-a3f6-3df10ea61574","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5de24915-8dec-49b6-8b16-f3170197c591","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ac054af3-d589-4565-9644-e6af75975c53","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3ec4f045-c714-4a22-bd1a-e07330b45ab3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"cddb9867-8e30-4951-9b13-3d1e78b9c45a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dc6d9911-785d-432f-8b53-56a1ccab302f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"77c88437-b587-496e-8b4d-1824ea8b5305","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ce963d96-5a8e-44c3-99de-c0dbebe234ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:46:14.614 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:14.631+0000 7f9de46ef640 10 librados: watch_flush enter 2024-03-29T10:46:14.614 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:14.631+0000 7f9de46ef640 10 librados: watch_flush exit 2024-03-29T10:46:14.614 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:14.631+0000 7f9de46ef640 20 client.4423.objecter shutdown clearing up homeless session... 2024-03-29T10:46:14.614 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:14.631+0000 7f9de46ef640 10 client.4423.objecter successfully canceled tick 2024-03-29T10:46:14.615 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:14.632+0000 7f9de46ef640 1 -- 172.21.15.88:0/1235031096 >> v1:172.21.15.88:6816/1752861349 conn(0x7f9dc006e250 legacy=0x7f9dc0070710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:14.615 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:14.632+0000 7f9de46ef640 10 monclient: shutdown 2024-03-29T10:46:14.615 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:14.632+0000 7f9de46ef640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:46:14.615 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:14.632+0000 7f9de46ef640 1 -- 172.21.15.88:0/1235031096 >> v1:172.21.15.196:6789/0 conn(0x7f9ddc0b17d0 legacy=0x7f9ddc12e260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:14.615 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:14.632+0000 7f9de46ef640 1 -- 172.21.15.88:0/1235031096 shutdown_connections 2024-03-29T10:46:14.615 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:14.632+0000 7f9de46ef640 1 -- 172.21.15.88:0/1235031096 >> 172.21.15.88:0/1235031096 conn(0x7f9ddc066a00 msgr2=0x7f9ddc13c410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:46:14.616 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:14.633+0000 7f9de46ef640 1 -- 172.21.15.88:0/1235031096 shutdown_connections 2024-03-29T10:46:14.616 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:14.633+0000 7f9de46ef640 1 -- 172.21.15.88:0/1235031096 wait complete. 2024-03-29T10:46:14.616 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:14.633+0000 7f9de46ef640 1 librados: shutdown 2024-03-29T10:46:14.632 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:46:21.439 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:46:21.592 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:21.610+0000 7f0536f65640 10 monclient: get_monmap_and_config 2024-03-29T10:46:21.593 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:21.610+0000 7f0536f65640 10 monclient: build_initial_monmap 2024-03-29T10:46:21.593 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:21.610+0000 7f0536f65640 10 monclient: monmap: 2024-03-29T10:46:21.593 INFO:teuthology.misc.health.smithi088.stderr:epoch 0 2024-03-29T10:46:21.593 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:46:21.593 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:46:21.611365+0000 2024-03-29T10:46:21.593 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:46:21.611365+0000 2024-03-29T10:46:21.593 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 0 (unknown) 2024-03-29T10:46:21.593 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:46:21.593 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-03-29T10:46:21.593 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-29T10:46:21.593 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-b 2024-03-29T10:46:21.594 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:46:21.594 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:21.612+0000 7f0536f65640 1 Processor -- start 2024-03-29T10:46:21.594 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:21.612+0000 7f0536f65640 1 -- start start 2024-03-29T10:46:21.594 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:21.612+0000 7f0536f65640 10 monclient: init 2024-03-29T10:46:21.595 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:21.613+0000 7f0536f65640 10 monclient: _reopen_session rank -1 2024-03-29T10:46:21.595 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:21.613+0000 7f0536f65640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:46:21.596 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:21.613+0000 7f0536f65640 10 monclient(hunting): picked mon.noname-c con 0x7f05300b0860 addr v1:172.21.15.196:6789/0 2024-03-29T10:46:21.596 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:21.613+0000 7f0536f65640 10 monclient(hunting): picked mon.noname-a con 0x7f053013ef30 addr v1:172.21.15.88:6789/0 2024-03-29T10:46:21.596 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:21.613+0000 7f0536f65640 10 monclient(hunting): picked mon.noname-b con 0x7f05300af8e0 addr v1:172.21.15.88:6790/0 2024-03-29T10:46:21.596 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:21.613+0000 7f0536f65640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05300b0c40 con 0x7f053013ef30 2024-03-29T10:46:21.596 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:21.613+0000 7f0536f65640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05300aed10 con 0x7f05300b0860 2024-03-29T10:46:21.596 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:21.613+0000 7f0536f65640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05300afd50 con 0x7f05300af8e0 2024-03-29T10:46:21.596 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:21.613+0000 7f0536f65640 10 monclient(hunting): _renew_subs 2024-03-29T10:46:21.596 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:21.613+0000 7f0536f65640 10 monclient(hunting): authenticate will time out at 1103.849278s 2024-03-29T10:46:21.599 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:21.616+0000 7f052ffff640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f053013ef30 0x7f05300af1d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:39960/0 (socket says 172.21.15.88:39960) 2024-03-29T10:46:21.599 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:21.616+0000 7f052ffff640 1 -- 172.21.15.88:0/540357724 learned_addr learned my addr 172.21.15.88:0/540357724 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-29T10:46:21.601 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:21.618+0000 7f052f7fe640 1 -- 172.21.15.88:0/540357724 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3058286245 0 0) 0x7f05300afd50 con 0x7f05300af8e0 2024-03-29T10:46:21.601 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:21.618+0000 7f052f7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:46:21.601 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:21.618+0000 7f052f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:21.601 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:21.618+0000 7f052f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:21.601 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:21.618+0000 7f052f7fe640 10 monclient(hunting): my global_id is 4454 2024-03-29T10:46:21.601 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:21.618+0000 7f052f7fe640 1 -- 172.21.15.88:0/540357724 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0520003520 con 0x7f05300af8e0 2024-03-29T10:46:21.601 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:21.618+0000 7f052f7fe640 1 -- 172.21.15.88:0/540357724 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1398236845 0 0) 0x7f05300aed10 con 0x7f05300b0860 2024-03-29T10:46:21.602 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:21.619+0000 7f052f7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:46:21.602 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:21.619+0000 7f052f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:21.602 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:21.619+0000 7f052f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:21.602 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:21.619+0000 7f052f7fe640 10 monclient(hunting): my global_id is 4426 2024-03-29T10:46:21.602 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:21.619+0000 7f052f7fe640 1 -- 172.21.15.88:0/540357724 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f05300afd50 con 0x7f05300b0860 2024-03-29T10:46:21.603 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:21.619+0000 7f052f7fe640 1 -- 172.21.15.88:0/540357724 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1293748508 0 0) 0x7f05300afd50 con 0x7f05300b0860 2024-03-29T10:46:21.603 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:21.619+0000 7f052f7fe640 1 -- 172.21.15.88:0/540357724 >> v1:172.21.15.88:6790/0 conn(0x7f05300af8e0 legacy=0x7f0530149a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:21.603 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:21.619+0000 7f052f7fe640 1 -- 172.21.15.88:0/540357724 >> v1:172.21.15.88:6789/0 conn(0x7f053013ef30 legacy=0x7f05300af1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:21.603 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:21.619+0000 7f052f7fe640 10 monclient: _finish_hunting 0 2024-03-29T10:46:21.603 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:21.619+0000 7f052f7fe640 1 monclient: found mon.noname-c 2024-03-29T10:46:21.604 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:21.619+0000 7f052f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:46:21.604 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:21.619+0000 7f052f7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.196:6789/0 2024-03-29T10:46:21.604 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:21.619+0000 7f052f7fe640 1 -- 172.21.15.88:0/540357724 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05300ab1b0 con 0x7f05300b0860 2024-03-29T10:46:21.604 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:21.619+0000 7f052f7fe640 10 monclient: _finish_auth 0 2024-03-29T10:46:21.604 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:21.619+0000 7f052f7fe640 10 monclient: _check_auth_tickets 2024-03-29T10:46:21.604 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:21.619+0000 7f052f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:46:21.604 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:21.619+0000 7f052f7fe640 1 -- 172.21.15.88:0/540357724 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f051c001520 con 0x7f05300b0860 2024-03-29T10:46:21.604 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:21.619+0000 7f052f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:21.604 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:21.619+0000 7f052f7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:46:21.604 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:21.619+0000 7f052f7fe640 10 monclient: dump: 2024-03-29T10:46:21.604 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:46:21.604 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:46:21.604 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:46:21.604 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:46:21.604 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:21.605 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:46:21.605 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:46:21.605 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:46:21.605 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:46:21.605 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:46:21.605 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:21.619+0000 7f0536f65640 5 monclient: authenticate success, global_id 4426 2024-03-29T10:46:21.605 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:21.619+0000 7f0536f65640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:46:21.605 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:21.619+0000 7f052f7fe640 1 -- 172.21.15.88:0/540357724 <== mon.1 v1:172.21.15.196:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f051c001680 con 0x7f05300b0860 2024-03-29T10:46:21.605 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:21.619+0000 7f052f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:46:21.605 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:21.619+0000 7f052f7fe640 1 -- 172.21.15.88:0/540357724 <== mon.1 v1:172.21.15.196:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f051c001ad0 con 0x7f05300b0860 2024-03-29T10:46:21.607 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:21.619+0000 7f052f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:21.607 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:21.619+0000 7f052f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:46:21.607 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:21.619+0000 7f052f7fe640 10 monclient: dump: 2024-03-29T10:46:21.607 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:46:21.607 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:46:21.607 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:46:21.607 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:46:21.607 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:21.607 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:46:21.608 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:46:21.608 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:46:21.608 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:46:21.608 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:46:21.608 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:21.620+0000 7f0536f65640 10 monclient: get_monmap_and_config success 2024-03-29T10:46:21.608 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:21.620+0000 7f0536f65640 4 set_mon_vals no callback set 2024-03-29T10:46:21.608 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:21.621+0000 7f0536f65640 10 monclient: shutdown 2024-03-29T10:46:21.608 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:21.621+0000 7f0536f65640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:46:21.608 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:21.621+0000 7f0536f65640 1 -- 172.21.15.88:0/540357724 >> v1:172.21.15.196:6789/0 conn(0x7f05300b0860 legacy=0x7f053013e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:21.608 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:21.622+0000 7f0536f65640 1 -- 172.21.15.88:0/540357724 shutdown_connections 2024-03-29T10:46:21.608 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:21.622+0000 7f0536f65640 1 -- 172.21.15.88:0/540357724 >> 172.21.15.88:0/540357724 conn(0x7f053006b660 msgr2=0x7f053006ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:46:21.608 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:21.623+0000 7f0536f65640 1 -- 172.21.15.88:0/540357724 shutdown_connections 2024-03-29T10:46:21.608 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:21.623+0000 7f0536f65640 1 -- 172.21.15.88:0/540357724 wait complete. 2024-03-29T10:46:21.608 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:21.624+0000 7f0536f65640 10 monclient: build_initial_monmap 2024-03-29T10:46:21.609 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:21.624+0000 7f0536f65640 10 monclient: monmap: 2024-03-29T10:46:21.609 INFO:teuthology.misc.health.smithi088.stderr:epoch 0 2024-03-29T10:46:21.609 INFO:teuthology.misc.health.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:46:21.609 INFO:teuthology.misc.health.smithi088.stderr:last_changed 0.000000 2024-03-29T10:46:21.609 INFO:teuthology.misc.health.smithi088.stderr:created 0.000000 2024-03-29T10:46:21.609 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 0 (unknown) 2024-03-29T10:46:21.609 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:46:21.609 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-03-29T10:46:21.609 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-29T10:46:21.609 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-c 2024-03-29T10:46:21.609 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:46:21.611 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:21.625+0000 7f0536f65640 1 librados: starting msgr at 2024-03-29T10:46:21.611 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:21.625+0000 7f0536f65640 1 librados: starting objecter 2024-03-29T10:46:21.611 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:21.625+0000 7f0536f65640 1 Processor -- start 2024-03-29T10:46:21.611 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:21.625+0000 7f0536f65640 1 -- start start 2024-03-29T10:46:21.611 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:21.625+0000 7f0536f65640 1 librados: setting wanted keys 2024-03-29T10:46:21.611 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:21.625+0000 7f0536f65640 1 librados: calling monclient init 2024-03-29T10:46:21.611 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:21.625+0000 7f0536f65640 10 monclient: init 2024-03-29T10:46:21.611 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:21.627+0000 7f0536f65640 10 monclient: _reopen_session rank -1 2024-03-29T10:46:21.611 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:21.627+0000 7f0536f65640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:46:21.611 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:21.627+0000 7f0536f65640 10 monclient(hunting): picked mon.noname-b con 0x7f05300af8e0 addr v1:172.21.15.196:6789/0 2024-03-29T10:46:21.612 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:21.627+0000 7f0536f65640 10 monclient(hunting): picked mon.noname-a con 0x7f05300b0860 addr v1:172.21.15.88:6789/0 2024-03-29T10:46:21.613 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:21.627+0000 7f0536f65640 10 monclient(hunting): picked mon.noname-c con 0x7f053013ef30 addr v1:172.21.15.88:6790/0 2024-03-29T10:46:21.613 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:21.627+0000 7f0536f65640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0530136730 con 0x7f05300b0860 2024-03-29T10:46:21.613 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:21.627+0000 7f0536f65640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05300b1610 con 0x7f05300af8e0 2024-03-29T10:46:21.613 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:21.627+0000 7f0536f65640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0530134eb0 con 0x7f053013ef30 2024-03-29T10:46:21.613 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:21.627+0000 7f0536f65640 10 monclient(hunting): _renew_subs 2024-03-29T10:46:21.613 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:21.627+0000 7f0536f65640 10 monclient(hunting): authenticate will time out at 1103.863608s 2024-03-29T10:46:21.613 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:21.628+0000 7f052ffff640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f05300b0860 0x7f05301d6720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:45616/0 (socket says 172.21.15.88:45616) 2024-03-29T10:46:21.613 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:21.628+0000 7f052ffff640 1 -- 172.21.15.88:0/4030351944 learned_addr learned my addr 172.21.15.88:0/4030351944 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-29T10:46:21.613 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:21.628+0000 7f052d7fa640 10 client.?.objecter ms_handle_connect 0x7f05300b0860 2024-03-29T10:46:21.613 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:21.628+0000 7f052d7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:46:21.613 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:21.628+0000 7f052d7fa640 1 -- 172.21.15.88:0/4030351944 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 885481678 0 0) 0x7f0530136730 con 0x7f05300b0860 2024-03-29T10:46:21.613 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:21.628+0000 7f052d7fa640 10 monclient(hunting): _negotiate 2024-03-29T10:46:21.613 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:21.628+0000 7f052d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:21.614 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:21.628+0000 7f052d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:21.614 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:21.628+0000 7f052d7fa640 10 monclient(hunting): my global_id is 4428 2024-03-29T10:46:21.614 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:21.628+0000 7f052d7fa640 1 -- 172.21.15.88:0/4030351944 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0514004eb0 con 0x7f05300b0860 2024-03-29T10:46:21.614 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:21.628+0000 7f052d7fa640 1 -- 172.21.15.88:0/4030351944 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1920029659 0 0) 0x7f0514004eb0 con 0x7f05300b0860 2024-03-29T10:46:21.614 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:21.628+0000 7f052d7fa640 1 -- 172.21.15.88:0/4030351944 --> v1:172.21.15.88:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0530136730 con 0x7f05300b0860 2024-03-29T10:46:21.614 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:21.628+0000 7f052d7fa640 1 -- 172.21.15.88:0/4030351944 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f05240017c0 con 0x7f05300b0860 2024-03-29T10:46:21.614 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:21.628+0000 7f052d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:21.614 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:21.628+0000 7f052d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:46:21.614 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:21.628+0000 7f052d7fa640 10 monclient(hunting): dump: 2024-03-29T10:46:21.614 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:46:21.614 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:46:21.614 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:46:21.614 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:46:21.614 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:21.615 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:46:21.615 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:46:21.615 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:46:21.615 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:46:21.615 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:46:21.615 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:21.628+0000 7f052d7fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:46:21.615 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:21.629+0000 7f052d7fa640 1 -- 172.21.15.88:0/4030351944 <== mon.0 v1:172.21.15.88:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2551985311 0 0) 0x7f0530136730 con 0x7f05300b0860 2024-03-29T10:46:21.615 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:21.629+0000 7f052d7fa640 1 -- 172.21.15.88:0/4030351944 >> v1:172.21.15.88:6790/0 conn(0x7f053013ef30 legacy=0x7f05301d6e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:46:21.615 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:21.630+0000 7f052d7fa640 1 -- 172.21.15.88:0/4030351944 >> v1:172.21.15.196:6789/0 conn(0x7f05300af8e0 legacy=0x7f05301347a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:46:21.617 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:21.632+0000 7f052d7fa640 10 monclient: _finish_hunting 0 2024-03-29T10:46:21.617 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:21.632+0000 7f052d7fa640 1 monclient: found mon.a 2024-03-29T10:46:21.618 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:21.632+0000 7f052d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:46:21.618 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:21.632+0000 7f052d7fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-03-29T10:46:21.618 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:21.632+0000 7f052d7fa640 1 -- 172.21.15.88:0/4030351944 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0530135040 con 0x7f05300b0860 2024-03-29T10:46:21.618 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:21.632+0000 7f052d7fa640 10 monclient: _finish_auth 0 2024-03-29T10:46:21.618 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:21.632+0000 7f052d7fa640 10 monclient: _check_auth_tickets 2024-03-29T10:46:21.618 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:21.632+0000 7f052d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:46:21.618 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:21.632+0000 7f0536f65640 5 monclient: authenticate success, global_id 4428 2024-03-29T10:46:21.618 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:21.632+0000 7f0536f65640 10 monclient: _renew_subs 2024-03-29T10:46:21.618 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:21.632+0000 7f0536f65640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-03-29T10:46:21.618 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:21.632+0000 7f0536f65640 1 -- 172.21.15.88:0/4030351944 --> v1:172.21.15.88:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0530136bc0 con 0x7f05300b0860 2024-03-29T10:46:21.618 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:21.632+0000 7f0536f65640 10 client.4428.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:46:21.618 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:21.632+0000 7f0536f65640 10 monclient: _renew_subs 2024-03-29T10:46:21.619 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:21.632+0000 7f0536f65640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-03-29T10:46:21.619 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:21.632+0000 7f0536f65640 1 -- 172.21.15.88:0/4030351944 --> v1:172.21.15.88:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0530137140 con 0x7f05300b0860 2024-03-29T10:46:21.619 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:21.632+0000 7f0536f65640 1 librados: init done 2024-03-29T10:46:21.619 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:21.634+0000 7f052d7fa640 1 -- 172.21.15.88:0/4030351944 <== mon.0 v1:172.21.15.88:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0524001d70 con 0x7f05300b0860 2024-03-29T10:46:21.619 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:21.634+0000 7f052d7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:46:21.619 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:21.634+0000 7f052d7fa640 1 -- 172.21.15.88:0/4030351944 <== mon.0 v1:172.21.15.88:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f05240017c0 con 0x7f05300b0860 2024-03-29T10:46:21.619 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:21.634+0000 7f052d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:21.619 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:21.634+0000 7f052d7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:46:21.619 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:21.634+0000 7f052d7fa640 10 monclient: dump: 2024-03-29T10:46:21.619 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:46:21.619 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:46:21.620 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:46:21.620 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:46:21.620 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:21.620 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:46:21.620 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:46:21.620 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:46:21.620 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:46:21.620 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:46:21.620 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:21.634+0000 7f052e7fc640 4 set_mon_vals no callback set 2024-03-29T10:46:21.621 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:21.637+0000 7f0536f65640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:46:21.621 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:21.637+0000 7f0536f65640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:46:21.621 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:21.637+0000 7f0536f65640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-03-29T10:46:21.621 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:21.637+0000 7f0536f65640 1 -- 172.21.15.88:0/4030351944 --> v1:172.21.15.88:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0510005080 con 0x7f05300b0860 2024-03-29T10:46:21.621 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:21.638+0000 7f052d7fa640 1 -- 172.21.15.88:0/4030351944 <== mon.0 v1:172.21.15.88:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4235629220 0 0) 0x7f0524004860 con 0x7f05300b0860 2024-03-29T10:46:21.621 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:21.638+0000 7f052d7fa640 1 -- 172.21.15.88:0/4030351944 <== mon.0 v1:172.21.15.88:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4257673582 0 0) 0x7f05240545b0 con 0x7f05300b0860 2024-03-29T10:46:21.621 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:21.638+0000 7f052d7fa640 10 client.4428.objecter ms_dispatch 0x7f0530000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:46:21.621 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:21.638+0000 7f052d7fa640 3 client.4428.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:46:21.621 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:21.638+0000 7f052d7fa640 3 client.4428.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:46:21.621 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:21.638+0000 7f052d7fa640 20 client.4428.objecter dump_active .. 0 homeless 2024-03-29T10:46:21.622 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:21.638+0000 7f0534d01640 10 monclient: get_auth_request con 0x7f05140765c0 auth_method 0 2024-03-29T10:46:21.622 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:21.639+0000 7f052d7fa640 10 client.4428.objecter ms_handle_connect 0x7f05140765c0 2024-03-29T10:46:21.630 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:21.647+0000 7f052d7fa640 1 -- 172.21.15.88:0/4030351944 <== mon.0 v1:172.21.15.88:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0524050850 con 0x7f05300b0860 2024-03-29T10:46:21.630 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:21.647+0000 7f052d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:46:21.630 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:21.647+0000 7f052d7fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:46:21.959 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:21.976+0000 7f0536f65640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:46:21.959 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:21.976+0000 7f0536f65640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:46:21.959 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:21.976+0000 7f0536f65640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-03-29T10:46:21.959 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:21.976+0000 7f0536f65640 1 -- 172.21.15.88:0/4030351944 --> v1:172.21.15.88:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0510005330 con 0x7f05300b0860 2024-03-29T10:46:21.959 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:21.977+0000 7f052d7fa640 1 -- 172.21.15.88:0/4030351944 <== mon.0 v1:172.21.15.88:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 344506374) 0x7f0524050850 con 0x7f05300b0860 2024-03-29T10:46:21.959 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:21.977+0000 7f052d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:46:21.960 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:21.977+0000 7f052d7fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:46:21.960 INFO:teuthology.misc.health.smithi088.stdout: 2024-03-29T10:46:21.960 INFO:teuthology.misc.health.smithi088.stdout:{"epoch":10,"fsid":"869f5188-f92b-42dd-a46d-8c646e0388a9","created":"2024-03-29T10:40:40.285087+0000","modified":"2024-03-29T10:40:42.536230+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:40:42.536230+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a26c0c1c-8de0-438c-a3f6-3df10ea61574","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5de24915-8dec-49b6-8b16-f3170197c591","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ac054af3-d589-4565-9644-e6af75975c53","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3ec4f045-c714-4a22-bd1a-e07330b45ab3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"cddb9867-8e30-4951-9b13-3d1e78b9c45a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dc6d9911-785d-432f-8b53-56a1ccab302f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"77c88437-b587-496e-8b4d-1824ea8b5305","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ce963d96-5a8e-44c3-99de-c0dbebe234ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:46:21.963 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:21.980+0000 7f0536f65640 10 librados: watch_flush enter 2024-03-29T10:46:21.963 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:21.980+0000 7f0536f65640 10 librados: watch_flush exit 2024-03-29T10:46:21.963 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:21.980+0000 7f0536f65640 20 client.4428.objecter shutdown clearing up homeless session... 2024-03-29T10:46:21.963 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:21.980+0000 7f0536f65640 10 client.4428.objecter successfully canceled tick 2024-03-29T10:46:21.963 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:21.980+0000 7f0536f65640 1 -- 172.21.15.88:0/4030351944 >> v1:172.21.15.88:6816/1752861349 conn(0x7f05140765c0 legacy=0x7f0514078a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:21.963 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:21.980+0000 7f0536f65640 10 monclient: shutdown 2024-03-29T10:46:21.963 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:21.980+0000 7f0536f65640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:46:21.963 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:21.980+0000 7f0536f65640 1 -- 172.21.15.88:0/4030351944 >> v1:172.21.15.88:6789/0 conn(0x7f05300b0860 legacy=0x7f05301d6720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:21.963 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:21.980+0000 7f0536f65640 1 -- 172.21.15.88:0/4030351944 shutdown_connections 2024-03-29T10:46:21.964 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:21.980+0000 7f0536f65640 1 -- 172.21.15.88:0/4030351944 >> 172.21.15.88:0/4030351944 conn(0x7f053006b660 msgr2=0x7f053013e5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:46:21.964 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:21.980+0000 7f0536f65640 1 -- 172.21.15.88:0/4030351944 shutdown_connections 2024-03-29T10:46:21.964 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:21.980+0000 7f0536f65640 1 -- 172.21.15.88:0/4030351944 wait complete. 2024-03-29T10:46:21.964 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:21.980+0000 7f0536f65640 1 librados: shutdown 2024-03-29T10:46:21.977 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:46:28.784 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:46:28.930 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:28.947+0000 7f69a92c1640 10 monclient: get_monmap_and_config 2024-03-29T10:46:28.930 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:28.947+0000 7f69a92c1640 10 monclient: build_initial_monmap 2024-03-29T10:46:28.930 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:28.947+0000 7f69a92c1640 10 monclient: monmap: 2024-03-29T10:46:28.930 INFO:teuthology.misc.health.smithi088.stderr:epoch 0 2024-03-29T10:46:28.930 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:46:28.930 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:46:28.948495+0000 2024-03-29T10:46:28.930 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:46:28.948495+0000 2024-03-29T10:46:28.930 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 0 (unknown) 2024-03-29T10:46:28.930 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:46:28.930 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-03-29T10:46:28.930 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-29T10:46:28.931 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-b 2024-03-29T10:46:28.931 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:46:28.933 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:28.950+0000 7f69a92c1640 1 Processor -- start 2024-03-29T10:46:28.933 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:28.951+0000 7f69a92c1640 1 -- start start 2024-03-29T10:46:28.933 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:28.951+0000 7f69a92c1640 10 monclient: init 2024-03-29T10:46:28.934 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:28.951+0000 7f69a92c1640 10 monclient: _reopen_session rank -1 2024-03-29T10:46:28.934 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:28.951+0000 7f69a92c1640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:46:28.934 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:28.951+0000 7f69a92c1640 10 monclient(hunting): picked mon.noname-a con 0x7f69a41457f0 addr v1:172.21.15.88:6789/0 2024-03-29T10:46:28.934 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:28.951+0000 7f69a92c1640 10 monclient(hunting): picked mon.noname-b con 0x7f69a413f650 addr v1:172.21.15.88:6790/0 2024-03-29T10:46:28.934 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:28.951+0000 7f69a92c1640 10 monclient(hunting): picked mon.noname-c con 0x7f69a40a9d70 addr v1:172.21.15.196:6789/0 2024-03-29T10:46:28.934 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:28.951+0000 7f69a92c1640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69a40aaef0 con 0x7f69a41457f0 2024-03-29T10:46:28.934 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:28.951+0000 7f69a92c1640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69a4145bd0 con 0x7f69a40a9d70 2024-03-29T10:46:28.935 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:28.951+0000 7f69a92c1640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69a4145eb0 con 0x7f69a413f650 2024-03-29T10:46:28.935 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:28.951+0000 7f69a92c1640 10 monclient(hunting): _renew_subs 2024-03-29T10:46:28.935 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:28.951+0000 7f69a92c1640 10 monclient(hunting): authenticate will time out at 1111.187878s 2024-03-29T10:46:28.938 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:28.955+0000 7f69a27fc640 1 --1- >> v1:172.21.15.88:6790/0 conn(0x7f69a413f650 0x7f69a40b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6790/0 says I am v1:172.21.15.88:33961/0 (socket says 172.21.15.88:33961) 2024-03-29T10:46:28.938 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:28.955+0000 7f69a27fc640 1 -- 172.21.15.88:0/3200895575 learned_addr learned my addr 172.21.15.88:0/3200895575 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-29T10:46:28.938 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:28.955+0000 7f69a1ffb640 1 -- 172.21.15.88:0/3200895575 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1513996160 0 0) 0x7f69a4145eb0 con 0x7f69a413f650 2024-03-29T10:46:28.938 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:28.955+0000 7f69a1ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:46:28.938 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:28.955+0000 7f69a1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:28.939 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:28.955+0000 7f69a1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:28.939 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:28.955+0000 7f69a1ffb640 10 monclient(hunting): my global_id is 4457 2024-03-29T10:46:28.939 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:28.955+0000 7f69a1ffb640 1 -- 172.21.15.88:0/3200895575 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f698c003520 con 0x7f69a413f650 2024-03-29T10:46:28.939 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:28.956+0000 7f69a1ffb640 1 -- 172.21.15.88:0/3200895575 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3218051230 0 0) 0x7f698c003520 con 0x7f69a413f650 2024-03-29T10:46:28.939 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:28.956+0000 7f69a1ffb640 1 -- 172.21.15.88:0/3200895575 >> v1:172.21.15.196:6789/0 conn(0x7f69a40a9d70 legacy=0x7f69a414d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:28.939 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:28.956+0000 7f69a1ffb640 1 -- 172.21.15.88:0/3200895575 >> v1:172.21.15.88:6789/0 conn(0x7f69a41457f0 legacy=0x7f69a413ef40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:46:28.941 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:28.957+0000 7f69a1ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:46:28.941 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:28.957+0000 7f69a1ffb640 1 monclient: found mon.noname-b 2024-03-29T10:46:28.941 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:28.957+0000 7f69a1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:46:28.941 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:28.957+0000 7f69a1ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.88:6790/0 2024-03-29T10:46:28.941 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:28.957+0000 7f69a1ffb640 1 -- 172.21.15.88:0/3200895575 --> v1:172.21.15.88:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69a4146340 con 0x7f69a413f650 2024-03-29T10:46:28.941 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:28.957+0000 7f69a1ffb640 10 monclient: _finish_auth 0 2024-03-29T10:46:28.941 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:28.957+0000 7f69a1ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:46:28.941 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:28.957+0000 7f69a1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:46:28.941 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:28.957+0000 7f69a92c1640 5 monclient: authenticate success, global_id 4457 2024-03-29T10:46:28.941 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:28.957+0000 7f69a92c1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:46:28.941 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:28.957+0000 7f69a1ffb640 1 -- 172.21.15.88:0/3200895575 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f6998001690 con 0x7f69a413f650 2024-03-29T10:46:28.941 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:28.957+0000 7f69a1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:28.942 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:28.957+0000 7f69a1ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:46:28.942 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:28.957+0000 7f69a1ffb640 10 monclient: dump: 2024-03-29T10:46:28.942 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:46:28.942 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:46:28.942 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:46:28.942 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:46:28.942 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:28.942 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:46:28.942 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:46:28.942 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:46:28.942 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:46:28.942 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:46:28.942 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:28.957+0000 7f69a92c1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:46:28.943 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:28.961+0000 7f69a1ffb640 1 -- 172.21.15.88:0/3200895575 <== mon.2 v1:172.21.15.88:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6998001960 con 0x7f69a413f650 2024-03-29T10:46:28.943 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:28.961+0000 7f69a1ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:46:28.943 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:28.961+0000 7f69a92c1640 10 monclient: get_monmap_and_config success 2024-03-29T10:46:28.943 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:28.961+0000 7f69a92c1640 4 set_mon_vals no callback set 2024-03-29T10:46:28.944 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:28.961+0000 7f69a1ffb640 1 -- 172.21.15.88:0/3200895575 <== mon.2 v1:172.21.15.88:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f6998001cf0 con 0x7f69a413f650 2024-03-29T10:46:28.944 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:28.961+0000 7f69a1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:28.944 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:28.961+0000 7f69a1ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:46:28.944 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:28.961+0000 7f69a1ffb640 10 monclient: dump: 2024-03-29T10:46:28.944 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:46:28.944 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:46:28.944 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:46:28.944 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:46:28.944 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:28.944 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:46:28.945 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:46:28.945 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:46:28.945 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:46:28.945 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:46:28.945 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:28.962+0000 7f69a92c1640 10 monclient: shutdown 2024-03-29T10:46:28.945 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:28.962+0000 7f69a92c1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:46:28.945 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:28.962+0000 7f69a92c1640 1 -- 172.21.15.88:0/3200895575 >> v1:172.21.15.88:6790/0 conn(0x7f69a413f650 legacy=0x7f69a40b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:28.945 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:28.962+0000 7f69a92c1640 1 -- 172.21.15.88:0/3200895575 shutdown_connections 2024-03-29T10:46:28.945 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:28.962+0000 7f69a92c1640 1 -- 172.21.15.88:0/3200895575 >> 172.21.15.88:0/3200895575 conn(0x7f69a4066a00 msgr2=0x7f69a4066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:46:28.946 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:28.962+0000 7f69a92c1640 1 -- 172.21.15.88:0/3200895575 shutdown_connections 2024-03-29T10:46:28.946 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:28.962+0000 7f69a92c1640 1 -- 172.21.15.88:0/3200895575 wait complete. 2024-03-29T10:46:28.946 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:28.963+0000 7f69a92c1640 10 monclient: build_initial_monmap 2024-03-29T10:46:28.946 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:28.963+0000 7f69a92c1640 10 monclient: monmap: 2024-03-29T10:46:28.946 INFO:teuthology.misc.health.smithi088.stderr:epoch 0 2024-03-29T10:46:28.946 INFO:teuthology.misc.health.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:46:28.946 INFO:teuthology.misc.health.smithi088.stderr:last_changed 0.000000 2024-03-29T10:46:28.946 INFO:teuthology.misc.health.smithi088.stderr:created 0.000000 2024-03-29T10:46:28.946 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 0 (unknown) 2024-03-29T10:46:28.947 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:46:28.947 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-03-29T10:46:28.947 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-29T10:46:28.947 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-c 2024-03-29T10:46:28.947 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:46:28.947 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:28.964+0000 7f69a92c1640 1 librados: starting msgr at 2024-03-29T10:46:28.947 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:28.964+0000 7f69a92c1640 1 librados: starting objecter 2024-03-29T10:46:28.947 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:28.964+0000 7f69a92c1640 1 Processor -- start 2024-03-29T10:46:28.948 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:28.964+0000 7f69a92c1640 1 -- start start 2024-03-29T10:46:28.948 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:28.964+0000 7f69a92c1640 1 librados: setting wanted keys 2024-03-29T10:46:28.948 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:28.964+0000 7f69a92c1640 1 librados: calling monclient init 2024-03-29T10:46:28.948 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:28.964+0000 7f69a92c1640 10 monclient: init 2024-03-29T10:46:28.951 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:28.965+0000 7f69a92c1640 10 monclient: _reopen_session rank -1 2024-03-29T10:46:28.951 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:28.965+0000 7f69a92c1640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:46:28.951 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:28.965+0000 7f69a92c1640 10 monclient(hunting): picked mon.noname-b con 0x7f69a40a9d70 addr v1:172.21.15.196:6789/0 2024-03-29T10:46:28.951 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:28.965+0000 7f69a92c1640 10 monclient(hunting): picked mon.noname-c con 0x7f69a413f650 addr v1:172.21.15.88:6790/0 2024-03-29T10:46:28.951 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:28.965+0000 7f69a92c1640 10 monclient(hunting): picked mon.noname-a con 0x7f69a41457f0 addr v1:172.21.15.88:6789/0 2024-03-29T10:46:28.951 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:28.965+0000 7f69a92c1640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69a4137a20 con 0x7f69a41457f0 2024-03-29T10:46:28.951 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:28.965+0000 7f69a92c1640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69a40ac8f0 con 0x7f69a40a9d70 2024-03-29T10:46:28.951 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:28.965+0000 7f69a92c1640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69a4142080 con 0x7f69a413f650 2024-03-29T10:46:28.951 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:28.965+0000 7f69a92c1640 10 monclient(hunting): _renew_subs 2024-03-29T10:46:28.951 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:28.965+0000 7f69a92c1640 10 monclient(hunting): authenticate will time out at 1111.201178s 2024-03-29T10:46:28.956 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:28.973+0000 7f69a27fc640 1 --1- >> v1:172.21.15.88:6790/0 conn(0x7f69a413f650 0x7f69a4136c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6790/0 says I am v1:172.21.15.88:34077/0 (socket says 172.21.15.88:34077) 2024-03-29T10:46:28.956 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:28.973+0000 7f69a27fc640 1 -- 172.21.15.88:0/415038328 learned_addr learned my addr 172.21.15.88:0/415038328 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-29T10:46:28.956 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:28.974+0000 7f698b7fe640 10 client.?.objecter ms_handle_connect 0x7f69a413f650 2024-03-29T10:46:28.957 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:28.974+0000 7f698b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:46:28.957 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:28.974+0000 7f698b7fe640 1 -- 172.21.15.88:0/415038328 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1335842004 0 0) 0x7f69a4142080 con 0x7f69a413f650 2024-03-29T10:46:28.957 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:28.974+0000 7f698b7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:46:28.957 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:28.974+0000 7f698b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:28.957 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:28.974+0000 7f698b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:28.957 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:28.974+0000 7f698b7fe640 10 monclient(hunting): my global_id is 4460 2024-03-29T10:46:28.957 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:28.974+0000 7f698b7fe640 1 -- 172.21.15.88:0/415038328 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6990004eb0 con 0x7f69a413f650 2024-03-29T10:46:28.957 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:28.975+0000 7f698b7fe640 1 -- 172.21.15.88:0/415038328 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1125839166 0 0) 0x7f6990004eb0 con 0x7f69a413f650 2024-03-29T10:46:28.958 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:28.975+0000 7f698b7fe640 1 -- 172.21.15.88:0/415038328 --> v1:172.21.15.88:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f69a4142080 con 0x7f69a413f650 2024-03-29T10:46:28.958 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:28.975+0000 7f698b7fe640 1 -- 172.21.15.88:0/415038328 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f6998001650 con 0x7f69a413f650 2024-03-29T10:46:28.958 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:28.975+0000 7f698b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:28.958 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:28.975+0000 7f698b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:46:28.958 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:28.975+0000 7f698b7fe640 10 monclient(hunting): dump: 2024-03-29T10:46:28.958 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:46:28.958 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:46:28.958 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:46:28.958 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:46:28.958 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:28.958 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:46:28.958 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:46:28.958 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:46:28.959 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:46:28.959 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:46:28.959 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:28.975+0000 7f698b7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:46:28.959 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:28.975+0000 7f698b7fe640 10 client.?.objecter ms_handle_connect 0x7f69a41457f0 2024-03-29T10:46:28.959 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:28.975+0000 7f698b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:46:28.959 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:28.975+0000 7f698b7fe640 10 client.?.objecter ms_handle_connect 0x7f69a40a9d70 2024-03-29T10:46:28.959 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:28.975+0000 7f698b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:46:28.959 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:28.975+0000 7f698b7fe640 1 -- 172.21.15.88:0/415038328 <== mon.2 v1:172.21.15.88:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2497609164 0 0) 0x7f69a4142080 con 0x7f69a413f650 2024-03-29T10:46:28.959 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:28.975+0000 7f698b7fe640 1 -- 172.21.15.88:0/415038328 >> v1:172.21.15.196:6789/0 conn(0x7f69a40a9d70 legacy=0x7f69a4142610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:28.959 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:28.975+0000 7f698b7fe640 1 -- 172.21.15.88:0/415038328 >> v1:172.21.15.88:6789/0 conn(0x7f69a41457f0 legacy=0x7f69a4137310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:28.959 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:28.975+0000 7f698b7fe640 10 monclient: _finish_hunting 0 2024-03-29T10:46:28.959 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:28.975+0000 7f698b7fe640 1 monclient: found mon.c 2024-03-29T10:46:28.959 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:28.975+0000 7f698b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:46:28.959 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:28.975+0000 7f698b7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-03-29T10:46:28.960 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:28.975+0000 7f698b7fe640 1 -- 172.21.15.88:0/415038328 --> v1:172.21.15.88:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69a4142d20 con 0x7f69a413f650 2024-03-29T10:46:28.960 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:28.975+0000 7f698b7fe640 10 monclient: _finish_auth 0 2024-03-29T10:46:28.960 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:28.975+0000 7f698b7fe640 10 monclient: _check_auth_tickets 2024-03-29T10:46:28.960 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:28.975+0000 7f698b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:46:28.961 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:28.975+0000 7f698b7fe640 1 -- 172.21.15.88:0/415038328 <== mon.2 v1:172.21.15.88:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6998004ca0 con 0x7f69a413f650 2024-03-29T10:46:28.962 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:28.975+0000 7f698b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:46:28.962 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:28.975+0000 7f698b7fe640 1 -- 172.21.15.88:0/415038328 <== mon.2 v1:172.21.15.88:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f6998005070 con 0x7f69a413f650 2024-03-29T10:46:28.962 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:28.975+0000 7f698b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:28.962 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:28.975+0000 7f698b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:46:28.962 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:28.975+0000 7f698b7fe640 10 monclient: dump: 2024-03-29T10:46:28.962 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:46:28.962 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:46:28.962 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:46:28.962 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:46:28.962 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:28.962 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:46:28.963 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:46:28.963 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:46:28.963 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:46:28.963 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:46:28.963 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:28.976+0000 7f69a17fa640 4 set_mon_vals no callback set 2024-03-29T10:46:28.963 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:28.977+0000 7f69a92c1640 5 monclient: authenticate success, global_id 4460 2024-03-29T10:46:28.963 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:28.977+0000 7f69a92c1640 10 monclient: _renew_subs 2024-03-29T10:46:28.963 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:28.977+0000 7f69a92c1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-03-29T10:46:28.963 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:28.977+0000 7f69a92c1640 1 -- 172.21.15.88:0/415038328 --> v1:172.21.15.88:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69a4141bf0 con 0x7f69a413f650 2024-03-29T10:46:28.963 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:28.977+0000 7f69a92c1640 10 client.4460.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:46:28.963 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:28.977+0000 7f69a92c1640 10 monclient: _renew_subs 2024-03-29T10:46:28.963 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:28.977+0000 7f69a92c1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-03-29T10:46:28.963 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:28.977+0000 7f69a92c1640 1 -- 172.21.15.88:0/415038328 --> v1:172.21.15.88:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f69a40ab800 con 0x7f69a413f650 2024-03-29T10:46:28.963 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:28.977+0000 7f69a92c1640 1 librados: init done 2024-03-29T10:46:28.964 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:28.978+0000 7f69a92c1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:46:28.964 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:28.978+0000 7f69a92c1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:46:28.964 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:28.978+0000 7f69a92c1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-03-29T10:46:28.964 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:28.979+0000 7f69a92c1640 1 -- 172.21.15.88:0/415038328 --> v1:172.21.15.88:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69a4142210 con 0x7f69a413f650 2024-03-29T10:46:28.964 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:28.981+0000 7f698b7fe640 1 -- 172.21.15.88:0/415038328 <== mon.2 v1:172.21.15.88:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4235629220 0 0) 0x7f69980051d0 con 0x7f69a413f650 2024-03-29T10:46:28.964 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:28.981+0000 7f698b7fe640 1 -- 172.21.15.88:0/415038328 <== mon.2 v1:172.21.15.88:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4257673582 0 0) 0x7f6998084c80 con 0x7f69a413f650 2024-03-29T10:46:28.964 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:28.981+0000 7f698b7fe640 10 client.4460.objecter ms_dispatch 0x7f69a4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:46:28.964 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:28.981+0000 7f698b7fe640 3 client.4460.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:46:28.965 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:28.981+0000 7f698b7fe640 3 client.4460.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:46:28.965 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:28.982+0000 7f698b7fe640 20 client.4460.objecter dump_active .. 0 homeless 2024-03-29T10:46:28.965 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:28.982+0000 7f69a2ffd640 10 monclient: get_auth_request con 0x7f699006ddc0 auth_method 0 2024-03-29T10:46:28.965 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:28.982+0000 7f698b7fe640 10 client.4460.objecter ms_handle_connect 0x7f699006ddc0 2024-03-29T10:46:28.971 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:28.988+0000 7f698b7fe640 1 -- 172.21.15.88:0/415038328 <== mon.2 v1:172.21.15.88:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6998084e60 con 0x7f69a413f650 2024-03-29T10:46:28.971 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:28.988+0000 7f698b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:46:28.971 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:28.988+0000 7f698b7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:46:29.302 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:29.320+0000 7f69a92c1640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:46:29.303 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:29.320+0000 7f69a92c1640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:46:29.303 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:29.320+0000 7f69a92c1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-03-29T10:46:29.303 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:29.320+0000 7f69a92c1640 1 -- 172.21.15.88:0/415038328 --> v1:172.21.15.88:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f69a40b3240 con 0x7f69a413f650 2024-03-29T10:46:29.305 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:29.322+0000 7f698b7fe640 1 -- 172.21.15.88:0/415038328 <== mon.2 v1:172.21.15.88:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 344506374) 0x7f6998050560 con 0x7f69a413f650 2024-03-29T10:46:29.305 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:29.322+0000 7f698b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:46:29.305 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:29.322+0000 7f698b7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:46:29.305 INFO:teuthology.misc.health.smithi088.stdout: 2024-03-29T10:46:29.305 INFO:teuthology.misc.health.smithi088.stdout:{"epoch":10,"fsid":"869f5188-f92b-42dd-a46d-8c646e0388a9","created":"2024-03-29T10:40:40.285087+0000","modified":"2024-03-29T10:40:42.536230+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:40:42.536230+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a26c0c1c-8de0-438c-a3f6-3df10ea61574","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5de24915-8dec-49b6-8b16-f3170197c591","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ac054af3-d589-4565-9644-e6af75975c53","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3ec4f045-c714-4a22-bd1a-e07330b45ab3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"cddb9867-8e30-4951-9b13-3d1e78b9c45a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dc6d9911-785d-432f-8b53-56a1ccab302f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"77c88437-b587-496e-8b4d-1824ea8b5305","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ce963d96-5a8e-44c3-99de-c0dbebe234ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:46:29.309 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:29.326+0000 7f69a92c1640 10 librados: watch_flush enter 2024-03-29T10:46:29.309 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:29.326+0000 7f69a92c1640 10 librados: watch_flush exit 2024-03-29T10:46:29.309 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:29.326+0000 7f69a92c1640 20 client.4460.objecter shutdown clearing up homeless session... 2024-03-29T10:46:29.309 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:29.326+0000 7f69a92c1640 10 client.4460.objecter successfully canceled tick 2024-03-29T10:46:29.310 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:29.327+0000 7f69a92c1640 1 -- 172.21.15.88:0/415038328 >> v1:172.21.15.88:6816/1752861349 conn(0x7f699006ddc0 legacy=0x7f6990070280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:29.310 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:29.327+0000 7f69a92c1640 10 monclient: shutdown 2024-03-29T10:46:29.310 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:29.327+0000 7f69a92c1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:46:29.310 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:29.327+0000 7f69a92c1640 1 -- 172.21.15.88:0/415038328 >> v1:172.21.15.88:6790/0 conn(0x7f69a413f650 legacy=0x7f69a4136c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:29.310 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:29.327+0000 7f69a92c1640 1 -- 172.21.15.88:0/415038328 shutdown_connections 2024-03-29T10:46:29.310 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:29.327+0000 7f69a92c1640 1 -- 172.21.15.88:0/415038328 >> 172.21.15.88:0/415038328 conn(0x7f69a4066a00 msgr2=0x7f69a413b340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:46:29.313 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:29.331+0000 7f69a92c1640 1 -- 172.21.15.88:0/415038328 shutdown_connections 2024-03-29T10:46:29.314 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:29.331+0000 7f69a92c1640 1 -- 172.21.15.88:0/415038328 wait complete. 2024-03-29T10:46:29.314 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:29.331+0000 7f69a92c1640 1 librados: shutdown 2024-03-29T10:46:29.335 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:46:36.142 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:46:36.292 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:36.309+0000 7fe3559d6640 10 monclient: get_monmap_and_config 2024-03-29T10:46:36.292 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:36.309+0000 7fe3559d6640 10 monclient: build_initial_monmap 2024-03-29T10:46:36.292 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:36.309+0000 7fe3559d6640 10 monclient: monmap: 2024-03-29T10:46:36.292 INFO:teuthology.misc.health.smithi088.stderr:epoch 0 2024-03-29T10:46:36.292 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:46:36.293 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:46:36.310260+0000 2024-03-29T10:46:36.293 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:46:36.310260+0000 2024-03-29T10:46:36.293 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 0 (unknown) 2024-03-29T10:46:36.293 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:46:36.293 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-03-29T10:46:36.293 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-29T10:46:36.293 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-b 2024-03-29T10:46:36.293 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:46:36.296 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:36.311+0000 7fe3559d6640 1 Processor -- start 2024-03-29T10:46:36.297 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:36.312+0000 7fe3559d6640 1 -- start start 2024-03-29T10:46:36.297 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:36.312+0000 7fe3559d6640 10 monclient: init 2024-03-29T10:46:36.297 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:36.313+0000 7fe3559d6640 10 monclient: _reopen_session rank -1 2024-03-29T10:46:36.297 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:36.313+0000 7fe3559d6640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:46:36.297 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:36.313+0000 7fe3559d6640 10 monclient(hunting): picked mon.noname-b con 0x7fe3500b0860 addr v1:172.21.15.88:6790/0 2024-03-29T10:46:36.297 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:36.313+0000 7fe3559d6640 10 monclient(hunting): picked mon.noname-c con 0x7fe35013ef30 addr v1:172.21.15.196:6789/0 2024-03-29T10:46:36.297 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:36.313+0000 7fe3559d6640 10 monclient(hunting): picked mon.noname-a con 0x7fe3500af8e0 addr v1:172.21.15.88:6789/0 2024-03-29T10:46:36.297 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:36.313+0000 7fe3559d6640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3500b0c40 con 0x7fe3500af8e0 2024-03-29T10:46:36.297 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:36.313+0000 7fe3559d6640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3500aed10 con 0x7fe35013ef30 2024-03-29T10:46:36.297 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:36.313+0000 7fe3559d6640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3500afd50 con 0x7fe3500b0860 2024-03-29T10:46:36.297 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:36.313+0000 7fe3559d6640 10 monclient(hunting): _renew_subs 2024-03-29T10:46:36.297 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:36.313+0000 7fe3559d6640 10 monclient(hunting): authenticate will time out at 1118.549204s 2024-03-29T10:46:36.301 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:36.318+0000 7fe34e7fc640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7fe35013ef30 0x7fe3500af1d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.88:47965/0 (socket says 172.21.15.88:47965) 2024-03-29T10:46:36.301 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:36.319+0000 7fe34e7fc640 1 -- 172.21.15.88:0/3673458493 learned_addr learned my addr 172.21.15.88:0/3673458493 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-29T10:46:36.302 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:36.320+0000 7fe34dffb640 1 -- 172.21.15.88:0/3673458493 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4028438808 0 0) 0x7fe3500aed10 con 0x7fe35013ef30 2024-03-29T10:46:36.302 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:36.320+0000 7fe34dffb640 10 monclient(hunting): _negotiate 2024-03-29T10:46:36.302 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:36.320+0000 7fe34dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:36.303 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:36.320+0000 7fe34dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:36.303 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:36.320+0000 7fe34dffb640 10 monclient(hunting): my global_id is 4435 2024-03-29T10:46:36.303 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:36.320+0000 7fe34dffb640 1 -- 172.21.15.88:0/3673458493 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe338003520 con 0x7fe35013ef30 2024-03-29T10:46:36.303 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:36.320+0000 7fe34dffb640 1 -- 172.21.15.88:0/3673458493 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 149415826 0 0) 0x7fe3500afd50 con 0x7fe3500b0860 2024-03-29T10:46:36.303 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:36.320+0000 7fe34dffb640 10 monclient(hunting): _negotiate 2024-03-29T10:46:36.303 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:36.320+0000 7fe34dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:36.304 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:36.320+0000 7fe34dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:36.304 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:36.320+0000 7fe34dffb640 10 monclient(hunting): my global_id is 4463 2024-03-29T10:46:36.304 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:36.320+0000 7fe34dffb640 1 -- 172.21.15.88:0/3673458493 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe3500aed10 con 0x7fe3500b0860 2024-03-29T10:46:36.304 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:36.320+0000 7fe34dffb640 1 -- 172.21.15.88:0/3673458493 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1491018451 0 0) 0x7fe3500b0c40 con 0x7fe3500af8e0 2024-03-29T10:46:36.304 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:36.320+0000 7fe34dffb640 10 monclient(hunting): _negotiate 2024-03-29T10:46:36.304 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:36.320+0000 7fe34dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:36.304 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:36.320+0000 7fe34dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:36.305 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:36.320+0000 7fe34dffb640 10 monclient(hunting): my global_id is 4434 2024-03-29T10:46:36.305 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:36.320+0000 7fe34dffb640 1 -- 172.21.15.88:0/3673458493 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe3500afd50 con 0x7fe3500af8e0 2024-03-29T10:46:36.306 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:36.320+0000 7fe34dffb640 1 -- 172.21.15.88:0/3673458493 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 190310051 0 0) 0x7fe3500afd50 con 0x7fe3500af8e0 2024-03-29T10:46:36.307 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:36.320+0000 7fe34dffb640 1 -- 172.21.15.88:0/3673458493 >> v1:172.21.15.88:6790/0 conn(0x7fe3500b0860 legacy=0x7fe35013e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:36.307 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:36.320+0000 7fe34dffb640 1 -- 172.21.15.88:0/3673458493 >> v1:172.21.15.196:6789/0 conn(0x7fe35013ef30 legacy=0x7fe3500af1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:36.307 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:36.320+0000 7fe34dffb640 10 monclient: _finish_hunting 0 2024-03-29T10:46:36.307 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:36.320+0000 7fe34dffb640 1 monclient: found mon.noname-a 2024-03-29T10:46:36.307 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:36.320+0000 7fe34dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:46:36.307 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:36.320+0000 7fe34dffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.88:6789/0 2024-03-29T10:46:36.308 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:36.321+0000 7fe34dffb640 1 -- 172.21.15.88:0/3673458493 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3500ab1b0 con 0x7fe3500af8e0 2024-03-29T10:46:36.308 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:36.321+0000 7fe34dffb640 10 monclient: _finish_auth 0 2024-03-29T10:46:36.308 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:36.321+0000 7fe34dffb640 10 monclient: _check_auth_tickets 2024-03-29T10:46:36.308 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:36.321+0000 7fe34dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:46:36.308 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:36.321+0000 7fe34dffb640 1 -- 172.21.15.88:0/3673458493 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7fe3480017e0 con 0x7fe3500af8e0 2024-03-29T10:46:36.308 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:36.321+0000 7fe34dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:36.309 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:36.321+0000 7fe34dffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:46:36.309 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:36.321+0000 7fe34dffb640 10 monclient: dump: 2024-03-29T10:46:36.309 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:46:36.309 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:46:36.309 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:46:36.309 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:46:36.309 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:36.310 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:46:36.310 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:46:36.310 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:46:36.310 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:46:36.310 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:46:36.310 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:36.321+0000 7fe34dffb640 1 -- 172.21.15.88:0/3673458493 <== mon.0 v1:172.21.15.88:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe348001940 con 0x7fe3500af8e0 2024-03-29T10:46:36.310 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:36.321+0000 7fe34dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:46:36.311 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:36.321+0000 7fe34dffb640 1 -- 172.21.15.88:0/3673458493 <== mon.0 v1:172.21.15.88:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7fe348001d50 con 0x7fe3500af8e0 2024-03-29T10:46:36.311 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:36.321+0000 7fe34dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:36.311 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:36.321+0000 7fe34dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:46:36.311 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:36.321+0000 7fe34dffb640 10 monclient: dump: 2024-03-29T10:46:36.311 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:46:36.311 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:46:36.311 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:46:36.312 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:46:36.312 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:36.312 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:46:36.312 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:46:36.312 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:46:36.312 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:46:36.312 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:46:36.312 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:36.324+0000 7fe3559d6640 5 monclient: authenticate success, global_id 4434 2024-03-29T10:46:36.313 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:36.324+0000 7fe3559d6640 10 monclient: get_monmap_and_config success 2024-03-29T10:46:36.313 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:36.324+0000 7fe3559d6640 4 set_mon_vals no callback set 2024-03-29T10:46:36.316 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:36.324+0000 7fe3559d6640 10 monclient: shutdown 2024-03-29T10:46:36.316 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:36.324+0000 7fe3559d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:46:36.316 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:36.324+0000 7fe3559d6640 1 -- 172.21.15.88:0/3673458493 >> v1:172.21.15.88:6789/0 conn(0x7fe3500af8e0 legacy=0x7fe350149a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:36.316 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:36.325+0000 7fe3559d6640 1 -- 172.21.15.88:0/3673458493 shutdown_connections 2024-03-29T10:46:36.316 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:36.325+0000 7fe3559d6640 1 -- 172.21.15.88:0/3673458493 >> 172.21.15.88:0/3673458493 conn(0x7fe35006b660 msgr2=0x7fe35006ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:46:36.316 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:36.327+0000 7fe3559d6640 1 -- 172.21.15.88:0/3673458493 shutdown_connections 2024-03-29T10:46:36.317 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:36.327+0000 7fe3559d6640 1 -- 172.21.15.88:0/3673458493 wait complete. 2024-03-29T10:46:36.317 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:36.328+0000 7fe3559d6640 10 monclient: build_initial_monmap 2024-03-29T10:46:36.317 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:36.328+0000 7fe3559d6640 10 monclient: monmap: 2024-03-29T10:46:36.317 INFO:teuthology.misc.health.smithi088.stderr:epoch 0 2024-03-29T10:46:36.317 INFO:teuthology.misc.health.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:46:36.317 INFO:teuthology.misc.health.smithi088.stderr:last_changed 0.000000 2024-03-29T10:46:36.317 INFO:teuthology.misc.health.smithi088.stderr:created 0.000000 2024-03-29T10:46:36.317 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 0 (unknown) 2024-03-29T10:46:36.318 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:46:36.318 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-03-29T10:46:36.318 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-29T10:46:36.318 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-c 2024-03-29T10:46:36.318 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:46:36.318 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:36.328+0000 7fe3559d6640 1 librados: starting msgr at 2024-03-29T10:46:36.318 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:36.328+0000 7fe3559d6640 1 librados: starting objecter 2024-03-29T10:46:36.318 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:36.329+0000 7fe3559d6640 1 Processor -- start 2024-03-29T10:46:36.319 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:36.329+0000 7fe3559d6640 1 -- start start 2024-03-29T10:46:36.319 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:36.329+0000 7fe3559d6640 1 librados: setting wanted keys 2024-03-29T10:46:36.319 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:36.329+0000 7fe3559d6640 1 librados: calling monclient init 2024-03-29T10:46:36.319 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:36.329+0000 7fe3559d6640 10 monclient: init 2024-03-29T10:46:36.319 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:36.329+0000 7fe3559d6640 10 monclient: _reopen_session rank -1 2024-03-29T10:46:36.319 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:36.329+0000 7fe3559d6640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:46:36.323 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:36.331+0000 7fe3559d6640 10 monclient(hunting): picked mon.noname-b con 0x7fe3500af8e0 addr v1:172.21.15.196:6789/0 2024-03-29T10:46:36.323 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:36.331+0000 7fe3559d6640 10 monclient(hunting): picked mon.noname-c con 0x7fe3500b0860 addr v1:172.21.15.88:6790/0 2024-03-29T10:46:36.323 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:36.331+0000 7fe3559d6640 10 monclient(hunting): picked mon.noname-a con 0x7fe35013ef30 addr v1:172.21.15.88:6789/0 2024-03-29T10:46:36.323 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:36.331+0000 7fe3559d6640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe350139a00 con 0x7fe35013ef30 2024-03-29T10:46:36.323 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:36.331+0000 7fe3559d6640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3500b3cb0 con 0x7fe3500af8e0 2024-03-29T10:46:36.324 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:36.331+0000 7fe3559d6640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3500b16e0 con 0x7fe3500b0860 2024-03-29T10:46:36.324 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:36.331+0000 7fe3559d6640 10 monclient(hunting): _renew_subs 2024-03-29T10:46:36.324 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:36.331+0000 7fe3559d6640 10 monclient(hunting): authenticate will time out at 1118.567491s 2024-03-29T10:46:36.324 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:36.336+0000 7fe34e7fc640 1 --1- >> v1:172.21.15.88:6790/0 conn(0x7fe3500b0860 0x7fe3501326f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6790/0 says I am v1:172.21.15.88:50539/0 (socket says 172.21.15.88:50539) 2024-03-29T10:46:36.324 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:36.336+0000 7fe34e7fc640 1 -- 172.21.15.88:0/1325626614 learned_addr learned my addr 172.21.15.88:0/1325626614 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-29T10:46:36.324 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:36.336+0000 7fe337fff640 10 client.?.objecter ms_handle_connect 0x7fe35013ef30 2024-03-29T10:46:36.324 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:36.336+0000 7fe337fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:46:36.325 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:36.336+0000 7fe337fff640 10 client.?.objecter ms_handle_connect 0x7fe3500b0860 2024-03-29T10:46:36.325 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:36.336+0000 7fe337fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:46:36.325 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:36.336+0000 7fe337fff640 10 client.?.objecter ms_handle_connect 0x7fe3500af8e0 2024-03-29T10:46:36.325 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:36.336+0000 7fe337fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:46:36.325 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:36.336+0000 7fe337fff640 1 -- 172.21.15.88:0/1325626614 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1672634657 0 0) 0x7fe350139a00 con 0x7fe35013ef30 2024-03-29T10:46:36.325 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:36.336+0000 7fe337fff640 10 monclient(hunting): _negotiate 2024-03-29T10:46:36.325 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:36.336+0000 7fe337fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:36.325 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:36.336+0000 7fe337fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:36.326 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:36.336+0000 7fe337fff640 10 monclient(hunting): my global_id is 4437 2024-03-29T10:46:36.326 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:36.337+0000 7fe337fff640 1 -- 172.21.15.88:0/1325626614 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe33c004eb0 con 0x7fe35013ef30 2024-03-29T10:46:36.326 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:36.337+0000 7fe337fff640 1 -- 172.21.15.88:0/1325626614 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 361504358 0 0) 0x7fe3500b3cb0 con 0x7fe3500af8e0 2024-03-29T10:46:36.326 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:36.337+0000 7fe337fff640 10 monclient(hunting): _negotiate 2024-03-29T10:46:36.326 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:36.337+0000 7fe337fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:36.326 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:36.337+0000 7fe337fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:36.326 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:36.337+0000 7fe337fff640 10 monclient(hunting): my global_id is 4438 2024-03-29T10:46:36.327 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:36.337+0000 7fe337fff640 1 -- 172.21.15.88:0/1325626614 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe350139a00 con 0x7fe3500af8e0 2024-03-29T10:46:36.327 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:36.337+0000 7fe337fff640 1 -- 172.21.15.88:0/1325626614 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3050004891 0 0) 0x7fe3500b16e0 con 0x7fe3500b0860 2024-03-29T10:46:36.327 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:36.337+0000 7fe337fff640 10 monclient(hunting): _negotiate 2024-03-29T10:46:36.327 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:36.337+0000 7fe337fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:36.327 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:36.337+0000 7fe337fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:36.327 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:36.337+0000 7fe337fff640 10 monclient(hunting): my global_id is 4466 2024-03-29T10:46:36.327 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:36.337+0000 7fe337fff640 1 -- 172.21.15.88:0/1325626614 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe3500b3cb0 con 0x7fe3500b0860 2024-03-29T10:46:36.327 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:36.337+0000 7fe337fff640 1 -- 172.21.15.88:0/1325626614 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1518670225 0 0) 0x7fe350139a00 con 0x7fe3500af8e0 2024-03-29T10:46:36.327 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:36.337+0000 7fe337fff640 1 -- 172.21.15.88:0/1325626614 --> v1:172.21.15.196:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe3500b16e0 con 0x7fe3500af8e0 2024-03-29T10:46:36.327 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:36.337+0000 7fe337fff640 1 -- 172.21.15.88:0/1325626614 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7fe340001830 con 0x7fe3500af8e0 2024-03-29T10:46:36.327 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:36.337+0000 7fe337fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:36.327 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:36.337+0000 7fe337fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:46:36.327 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:36.337+0000 7fe337fff640 10 monclient(hunting): dump: 2024-03-29T10:46:36.328 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:46:36.328 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:46:36.328 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:46:36.328 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:46:36.328 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:36.328 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:46:36.328 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:46:36.328 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:46:36.328 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:46:36.328 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:46:36.328 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:36.337+0000 7fe337fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:46:36.328 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:36.337+0000 7fe337fff640 1 -- 172.21.15.88:0/1325626614 <== mon.1 v1:172.21.15.196:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3473103125 0 0) 0x7fe3500b16e0 con 0x7fe3500af8e0 2024-03-29T10:46:36.328 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:36.337+0000 7fe337fff640 1 -- 172.21.15.88:0/1325626614 >> v1:172.21.15.88:6790/0 conn(0x7fe3500b0860 legacy=0x7fe3501326f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:36.329 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:36.337+0000 7fe337fff640 1 -- 172.21.15.88:0/1325626614 >> v1:172.21.15.88:6789/0 conn(0x7fe35013ef30 legacy=0x7fe3501d31a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:36.329 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:36.337+0000 7fe337fff640 10 monclient: _finish_hunting 0 2024-03-29T10:46:36.329 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:36.337+0000 7fe337fff640 1 monclient: found mon.b 2024-03-29T10:46:36.329 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:36.337+0000 7fe337fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:46:36.329 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:36.337+0000 7fe337fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T10:46:36.329 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:36.337+0000 7fe337fff640 1 -- 172.21.15.88:0/1325626614 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3500b1870 con 0x7fe3500af8e0 2024-03-29T10:46:36.329 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:36.337+0000 7fe337fff640 10 monclient: _finish_auth 0 2024-03-29T10:46:36.329 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:36.337+0000 7fe337fff640 10 monclient: _check_auth_tickets 2024-03-29T10:46:36.329 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:36.337+0000 7fe337fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:46:36.329 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:36.338+0000 7fe337fff640 1 -- 172.21.15.88:0/1325626614 <== mon.1 v1:172.21.15.196:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe340001d60 con 0x7fe3500af8e0 2024-03-29T10:46:36.329 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:36.338+0000 7fe337fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:46:36.329 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:36.338+0000 7fe337fff640 1 -- 172.21.15.88:0/1325626614 <== mon.1 v1:172.21.15.196:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7fe3400037b0 con 0x7fe3500af8e0 2024-03-29T10:46:36.329 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:36.338+0000 7fe337fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:36.329 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:36.338+0000 7fe337fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:46:36.329 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:36.338+0000 7fe337fff640 10 monclient: dump: 2024-03-29T10:46:36.330 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:46:36.330 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:46:36.330 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:46:36.330 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:46:36.330 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:36.330 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:46:36.330 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:46:36.330 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:46:36.330 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:46:36.330 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:46:36.330 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:36.338+0000 7fe34d7fa640 4 set_mon_vals no callback set 2024-03-29T10:46:36.330 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:36.339+0000 7fe3559d6640 5 monclient: authenticate success, global_id 4438 2024-03-29T10:46:36.330 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:36.339+0000 7fe3559d6640 10 monclient: _renew_subs 2024-03-29T10:46:36.331 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:36.339+0000 7fe3559d6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T10:46:36.331 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:36.339+0000 7fe3559d6640 1 -- 172.21.15.88:0/1325626614 --> v1:172.21.15.196:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3500b40d0 con 0x7fe3500af8e0 2024-03-29T10:46:36.331 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:36.339+0000 7fe3559d6640 10 client.4438.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:46:36.331 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:36.339+0000 7fe3559d6640 10 monclient: _renew_subs 2024-03-29T10:46:36.331 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:36.339+0000 7fe3559d6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T10:46:36.331 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:36.339+0000 7fe3559d6640 1 -- 172.21.15.88:0/1325626614 --> v1:172.21.15.196:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe350132fe0 con 0x7fe3500af8e0 2024-03-29T10:46:36.331 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:36.339+0000 7fe3559d6640 1 librados: init done 2024-03-29T10:46:36.332 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:36.343+0000 7fe3559d6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:46:36.332 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:36.343+0000 7fe3559d6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:46:36.332 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:36.343+0000 7fe3559d6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T10:46:36.332 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:36.343+0000 7fe3559d6640 1 -- 172.21.15.88:0/1325626614 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3500ab750 con 0x7fe3500af8e0 2024-03-29T10:46:36.333 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:36.344+0000 7fe337fff640 1 -- 172.21.15.88:0/1325626614 <== mon.1 v1:172.21.15.196:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4235629220 0 0) 0x7fe340019780 con 0x7fe3500af8e0 2024-03-29T10:46:36.333 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:36.345+0000 7fe337fff640 1 -- 172.21.15.88:0/1325626614 <== mon.1 v1:172.21.15.196:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4257673582 0 0) 0x7fe3400836b0 con 0x7fe3500af8e0 2024-03-29T10:46:36.333 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:36.345+0000 7fe337fff640 10 client.4438.objecter ms_dispatch 0x7fe350000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:46:36.333 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:36.345+0000 7fe337fff640 3 client.4438.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:46:36.333 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:36.345+0000 7fe337fff640 3 client.4438.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:46:36.333 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:36.345+0000 7fe337fff640 20 client.4438.objecter dump_active .. 0 homeless 2024-03-29T10:46:36.333 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:36.346+0000 7fe34e7fc640 10 monclient: get_auth_request con 0x7fe33c06deb0 auth_method 0 2024-03-29T10:46:36.333 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:36.346+0000 7fe337fff640 10 client.4438.objecter ms_handle_connect 0x7fe33c06deb0 2024-03-29T10:46:36.334 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:36.352+0000 7fe337fff640 1 -- 172.21.15.88:0/1325626614 <== mon.1 v1:172.21.15.196:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe3500ab750 con 0x7fe3500af8e0 2024-03-29T10:46:36.334 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:36.352+0000 7fe337fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:46:36.335 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:36.352+0000 7fe337fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:46:36.670 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:36.687+0000 7fe3559d6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:46:36.670 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:36.687+0000 7fe3559d6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:46:36.670 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:36.687+0000 7fe3559d6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T10:46:36.670 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:36.687+0000 7fe3559d6640 1 -- 172.21.15.88:0/1325626614 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe3500ab750 con 0x7fe3500af8e0 2024-03-29T10:46:36.671 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:36.689+0000 7fe337fff640 1 -- 172.21.15.88:0/1325626614 <== mon.1 v1:172.21.15.196:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 344506374) 0x7fe3500ab750 con 0x7fe3500af8e0 2024-03-29T10:46:36.671 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:36.689+0000 7fe337fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:46:36.671 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:36.689+0000 7fe337fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:46:36.672 INFO:teuthology.misc.health.smithi088.stdout: 2024-03-29T10:46:36.672 INFO:teuthology.misc.health.smithi088.stdout:{"epoch":10,"fsid":"869f5188-f92b-42dd-a46d-8c646e0388a9","created":"2024-03-29T10:40:40.285087+0000","modified":"2024-03-29T10:40:42.536230+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:40:42.536230+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a26c0c1c-8de0-438c-a3f6-3df10ea61574","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5de24915-8dec-49b6-8b16-f3170197c591","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ac054af3-d589-4565-9644-e6af75975c53","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3ec4f045-c714-4a22-bd1a-e07330b45ab3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"cddb9867-8e30-4951-9b13-3d1e78b9c45a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dc6d9911-785d-432f-8b53-56a1ccab302f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"77c88437-b587-496e-8b4d-1824ea8b5305","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ce963d96-5a8e-44c3-99de-c0dbebe234ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:46:36.675 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:36.692+0000 7fe3559d6640 10 librados: watch_flush enter 2024-03-29T10:46:36.675 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:36.692+0000 7fe3559d6640 10 librados: watch_flush exit 2024-03-29T10:46:36.675 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:36.692+0000 7fe3559d6640 20 client.4438.objecter shutdown clearing up homeless session... 2024-03-29T10:46:36.675 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:36.692+0000 7fe3559d6640 10 client.4438.objecter successfully canceled tick 2024-03-29T10:46:36.675 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:36.692+0000 7fe3559d6640 1 -- 172.21.15.88:0/1325626614 >> v1:172.21.15.88:6816/1752861349 conn(0x7fe33c06deb0 legacy=0x7fe33c070370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:36.675 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:36.692+0000 7fe3559d6640 10 monclient: shutdown 2024-03-29T10:46:36.676 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:36.692+0000 7fe3559d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:46:36.676 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:36.692+0000 7fe3559d6640 1 -- 172.21.15.88:0/1325626614 >> v1:172.21.15.196:6789/0 conn(0x7fe3500af8e0 legacy=0x7fe3500b23f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:36.676 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:36.693+0000 7fe3559d6640 1 -- 172.21.15.88:0/1325626614 shutdown_connections 2024-03-29T10:46:36.676 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:36.693+0000 7fe3559d6640 1 -- 172.21.15.88:0/1325626614 >> 172.21.15.88:0/1325626614 conn(0x7fe35006b660 msgr2=0x7fe350134920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:46:36.679 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:36.696+0000 7fe3559d6640 1 -- 172.21.15.88:0/1325626614 shutdown_connections 2024-03-29T10:46:36.679 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:36.696+0000 7fe3559d6640 1 -- 172.21.15.88:0/1325626614 wait complete. 2024-03-29T10:46:36.679 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:36.696+0000 7fe3559d6640 1 librados: shutdown 2024-03-29T10:46:36.698 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:46:43.508 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:46:43.657 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:43.674+0000 7f40fb5ac640 10 monclient: get_monmap_and_config 2024-03-29T10:46:43.657 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:43.674+0000 7f40fb5ac640 10 monclient: build_initial_monmap 2024-03-29T10:46:43.657 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:43.674+0000 7f40fb5ac640 10 monclient: monmap: 2024-03-29T10:46:43.657 INFO:teuthology.misc.health.smithi088.stderr:epoch 0 2024-03-29T10:46:43.657 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:46:43.657 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:46:43.675622+0000 2024-03-29T10:46:43.658 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:46:43.675622+0000 2024-03-29T10:46:43.658 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 0 (unknown) 2024-03-29T10:46:43.658 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:46:43.658 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-03-29T10:46:43.658 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-29T10:46:43.658 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-b 2024-03-29T10:46:43.658 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:46:43.659 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:43.675+0000 7f40fb5ac640 1 Processor -- start 2024-03-29T10:46:43.659 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:43.675+0000 7f40fb5ac640 1 -- start start 2024-03-29T10:46:43.659 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:43.675+0000 7f40fb5ac640 10 monclient: init 2024-03-29T10:46:43.659 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:43.676+0000 7f40fb5ac640 10 monclient: _reopen_session rank -1 2024-03-29T10:46:43.660 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:43.676+0000 7f40fb5ac640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:46:43.660 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:43.676+0000 7f40fb5ac640 10 monclient(hunting): picked mon.noname-a con 0x7f40f40b1f60 addr v1:172.21.15.88:6789/0 2024-03-29T10:46:43.660 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:43.676+0000 7f40fb5ac640 10 monclient(hunting): picked mon.noname-c con 0x7f40f4071eb0 addr v1:172.21.15.196:6789/0 2024-03-29T10:46:43.660 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:43.676+0000 7f40fb5ac640 10 monclient(hunting): picked mon.noname-b con 0x7f40f4143d10 addr v1:172.21.15.88:6790/0 2024-03-29T10:46:43.660 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:43.676+0000 7f40fb5ac640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40f406ef80 con 0x7f40f40b1f60 2024-03-29T10:46:43.660 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:43.676+0000 7f40fb5ac640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40f406f260 con 0x7f40f4071eb0 2024-03-29T10:46:43.660 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:43.676+0000 7f40fb5ac640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40f40b40c0 con 0x7f40f4143d10 2024-03-29T10:46:43.660 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:43.676+0000 7f40fb5ac640 10 monclient(hunting): _renew_subs 2024-03-29T10:46:43.660 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:43.676+0000 7f40fb5ac640 10 monclient(hunting): authenticate will time out at 1125.912521s 2024-03-29T10:46:43.668 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:43.681+0000 7f40f9b49640 1 --1- >> v1:172.21.15.88:6790/0 conn(0x7f40f4143d10 0x7f40f4146b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6790/0 says I am v1:172.21.15.88:46305/0 (socket says 172.21.15.88:46305) 2024-03-29T10:46:43.668 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:43.681+0000 7f40f9b49640 1 -- 172.21.15.88:0/2128036574 learned_addr learned my addr 172.21.15.88:0/2128036574 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-29T10:46:43.668 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:43.682+0000 7f40e7fff640 1 -- 172.21.15.88:0/2128036574 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3235759088 0 0) 0x7f40f40b40c0 con 0x7f40f4143d10 2024-03-29T10:46:43.668 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:43.682+0000 7f40e7fff640 10 monclient(hunting): _negotiate 2024-03-29T10:46:43.668 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:43.682+0000 7f40e7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:43.668 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:43.682+0000 7f40e7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:43.668 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:43.682+0000 7f40e7fff640 10 monclient(hunting): my global_id is 4469 2024-03-29T10:46:43.668 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:43.682+0000 7f40e7fff640 1 -- 172.21.15.88:0/2128036574 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f40e0003520 con 0x7f40f4143d10 2024-03-29T10:46:43.668 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:43.682+0000 7f40e7fff640 1 -- 172.21.15.88:0/2128036574 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1274504556 0 0) 0x7f40e0003520 con 0x7f40f4143d10 2024-03-29T10:46:43.668 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:43.682+0000 7f40e7fff640 1 -- 172.21.15.88:0/2128036574 >> v1:172.21.15.196:6789/0 conn(0x7f40f4071eb0 legacy=0x7f40f4146470 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:46:43.669 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:43.682+0000 7f40e7fff640 1 -- 172.21.15.88:0/2128036574 >> v1:172.21.15.88:6789/0 conn(0x7f40f40b1f60 legacy=0x7f40f4143600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:43.669 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:43.682+0000 7f40e7fff640 10 monclient: _finish_hunting 0 2024-03-29T10:46:43.669 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:43.682+0000 7f40e7fff640 1 monclient: found mon.noname-b 2024-03-29T10:46:43.669 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:43.682+0000 7f40e7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:46:43.669 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:43.682+0000 7f40e7fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.88:6790/0 2024-03-29T10:46:43.669 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:43.682+0000 7f40e7fff640 1 -- 172.21.15.88:0/2128036574 --> v1:172.21.15.88:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40f4155d70 con 0x7f40f4143d10 2024-03-29T10:46:43.669 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:43.682+0000 7f40e7fff640 10 monclient: _finish_auth 0 2024-03-29T10:46:43.669 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:43.682+0000 7f40e7fff640 10 monclient: _check_auth_tickets 2024-03-29T10:46:43.669 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:43.682+0000 7f40e7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:46:43.669 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:43.682+0000 7f40fb5ac640 5 monclient: authenticate success, global_id 4469 2024-03-29T10:46:43.669 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:43.682+0000 7f40fb5ac640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:46:43.669 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:43.682+0000 7f40e7fff640 1 -- 172.21.15.88:0/2128036574 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f40ec001a10 con 0x7f40f4143d10 2024-03-29T10:46:43.669 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:43.682+0000 7f40e7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:43.669 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:43.682+0000 7f40e7fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:46:43.669 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:43.682+0000 7f40e7fff640 10 monclient: dump: 2024-03-29T10:46:43.670 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:46:43.670 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:46:43.670 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:46:43.670 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:46:43.670 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:43.670 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:46:43.670 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:46:43.670 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:46:43.670 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:46:43.670 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:46:43.670 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:43.682+0000 7f40fb5ac640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:46:43.670 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:43.683+0000 7f40e7fff640 1 -- 172.21.15.88:0/2128036574 <== mon.2 v1:172.21.15.88:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f40ec001ca0 con 0x7f40f4143d10 2024-03-29T10:46:43.670 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:43.683+0000 7f40e7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:46:43.670 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:43.683+0000 7f40e7fff640 1 -- 172.21.15.88:0/2128036574 <== mon.2 v1:172.21.15.88:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f40ec002030 con 0x7f40f4143d10 2024-03-29T10:46:43.671 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:43.683+0000 7f40e7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:43.671 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:43.683+0000 7f40e7fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:46:43.671 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:43.683+0000 7f40e7fff640 10 monclient: dump: 2024-03-29T10:46:43.671 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:46:43.671 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:46:43.671 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:46:43.671 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:46:43.671 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:43.671 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:46:43.671 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:46:43.671 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:46:43.671 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:46:43.671 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:46:43.671 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:43.683+0000 7f40fb5ac640 10 monclient: get_monmap_and_config success 2024-03-29T10:46:43.671 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:43.683+0000 7f40fb5ac640 4 set_mon_vals no callback set 2024-03-29T10:46:43.672 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:43.684+0000 7f40fb5ac640 10 monclient: shutdown 2024-03-29T10:46:43.672 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:43.684+0000 7f40fb5ac640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:46:43.672 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:43.684+0000 7f40fb5ac640 1 -- 172.21.15.88:0/2128036574 >> v1:172.21.15.88:6790/0 conn(0x7f40f4143d10 legacy=0x7f40f4146b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:43.672 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:43.688+0000 7f40fb5ac640 1 -- 172.21.15.88:0/2128036574 shutdown_connections 2024-03-29T10:46:43.672 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:43.688+0000 7f40fb5ac640 1 -- 172.21.15.88:0/2128036574 >> 172.21.15.88:0/2128036574 conn(0x7f40f4066a00 msgr2=0x7f40f4066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:46:43.674 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:43.691+0000 7f40fb5ac640 1 -- 172.21.15.88:0/2128036574 shutdown_connections 2024-03-29T10:46:43.677 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:43.692+0000 7f40fb5ac640 1 -- 172.21.15.88:0/2128036574 wait complete. 2024-03-29T10:46:43.678 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:43.693+0000 7f40fb5ac640 10 monclient: build_initial_monmap 2024-03-29T10:46:43.678 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:43.693+0000 7f40fb5ac640 10 monclient: monmap: 2024-03-29T10:46:43.678 INFO:teuthology.misc.health.smithi088.stderr:epoch 0 2024-03-29T10:46:43.678 INFO:teuthology.misc.health.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:46:43.678 INFO:teuthology.misc.health.smithi088.stderr:last_changed 0.000000 2024-03-29T10:46:43.678 INFO:teuthology.misc.health.smithi088.stderr:created 0.000000 2024-03-29T10:46:43.678 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 0 (unknown) 2024-03-29T10:46:43.678 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:46:43.678 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-03-29T10:46:43.678 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-29T10:46:43.678 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-c 2024-03-29T10:46:43.678 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:46:43.679 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:43.693+0000 7f40fb5ac640 1 librados: starting msgr at 2024-03-29T10:46:43.679 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:43.693+0000 7f40fb5ac640 1 librados: starting objecter 2024-03-29T10:46:43.679 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:43.693+0000 7f40fb5ac640 1 Processor -- start 2024-03-29T10:46:43.679 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:43.693+0000 7f40fb5ac640 1 -- start start 2024-03-29T10:46:43.679 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:43.693+0000 7f40fb5ac640 1 librados: setting wanted keys 2024-03-29T10:46:43.679 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:43.693+0000 7f40fb5ac640 1 librados: calling monclient init 2024-03-29T10:46:43.679 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:43.693+0000 7f40fb5ac640 10 monclient: init 2024-03-29T10:46:43.679 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:43.694+0000 7f40fb5ac640 10 monclient: _reopen_session rank -1 2024-03-29T10:46:43.679 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:43.694+0000 7f40fb5ac640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:46:43.679 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:43.694+0000 7f40fb5ac640 10 monclient(hunting): picked mon.noname-a con 0x7f40f40b1f60 addr v1:172.21.15.88:6789/0 2024-03-29T10:46:43.679 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:43.695+0000 7f40fb5ac640 10 monclient(hunting): picked mon.noname-b con 0x7f40f4143d10 addr v1:172.21.15.196:6789/0 2024-03-29T10:46:43.679 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:43.695+0000 7f40fb5ac640 10 monclient(hunting): picked mon.noname-c con 0x7f40f40708e0 addr v1:172.21.15.88:6790/0 2024-03-29T10:46:43.682 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:43.695+0000 7f40fb5ac640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40f41496c0 con 0x7f40f40b1f60 2024-03-29T10:46:43.682 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:43.695+0000 7f40fb5ac640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40f414af70 con 0x7f40f4143d10 2024-03-29T10:46:43.682 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:43.695+0000 7f40fb5ac640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40f414a310 con 0x7f40f40708e0 2024-03-29T10:46:43.682 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:43.695+0000 7f40fb5ac640 10 monclient(hunting): _renew_subs 2024-03-29T10:46:43.682 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:43.695+0000 7f40fb5ac640 10 monclient(hunting): authenticate will time out at 1125.931550s 2024-03-29T10:46:43.682 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:43.700+0000 7f40f8b47640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f40f4143d10 0x7f40f4149c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.88:44555/0 (socket says 172.21.15.88:44555) 2024-03-29T10:46:43.682 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:43.700+0000 7f40f8b47640 1 -- 172.21.15.88:0/3926515217 learned_addr learned my addr 172.21.15.88:0/3926515217 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-29T10:46:43.683 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:43.700+0000 7f40e5ffb640 10 client.?.objecter ms_handle_connect 0x7f40f40b1f60 2024-03-29T10:46:43.683 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:43.700+0000 7f40e5ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:46:43.684 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:43.701+0000 7f40e5ffb640 10 client.?.objecter ms_handle_connect 0x7f40f4143d10 2024-03-29T10:46:43.684 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:43.701+0000 7f40e5ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:46:43.684 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:43.701+0000 7f40e5ffb640 1 -- 172.21.15.88:0/3926515217 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2682924291 0 0) 0x7f40f41496c0 con 0x7f40f40b1f60 2024-03-29T10:46:43.684 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:43.701+0000 7f40e5ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:46:43.684 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:43.701+0000 7f40e5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:43.686 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:43.701+0000 7f40e5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:43.686 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:43.701+0000 7f40e5ffb640 10 monclient(hunting): my global_id is 4443 2024-03-29T10:46:43.686 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:43.701+0000 7f40e5ffb640 1 -- 172.21.15.88:0/3926515217 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f40d8004eb0 con 0x7f40f40b1f60 2024-03-29T10:46:43.686 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:43.701+0000 7f40e5ffb640 10 client.?.objecter ms_handle_connect 0x7f40f40708e0 2024-03-29T10:46:43.687 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:43.701+0000 7f40e5ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:46:43.687 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:43.703+0000 7f40e5ffb640 1 -- 172.21.15.88:0/3926515217 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1848074296 0 0) 0x7f40f414af70 con 0x7f40f4143d10 2024-03-29T10:46:43.687 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:43.703+0000 7f40e5ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:46:43.687 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:43.703+0000 7f40e5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:43.687 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:43.703+0000 7f40e5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:43.687 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:43.703+0000 7f40e5ffb640 10 monclient(hunting): my global_id is 4441 2024-03-29T10:46:43.687 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:43.703+0000 7f40e5ffb640 1 -- 172.21.15.88:0/3926515217 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f40d80051d0 con 0x7f40f4143d10 2024-03-29T10:46:43.687 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:43.705+0000 7f40e5ffb640 1 -- 172.21.15.88:0/3926515217 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3131768766 0 0) 0x7f40f414a310 con 0x7f40f40708e0 2024-03-29T10:46:43.687 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:43.705+0000 7f40e5ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:46:43.688 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:43.705+0000 7f40e5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:43.688 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:43.705+0000 7f40e5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:43.688 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:43.705+0000 7f40e5ffb640 10 monclient(hunting): my global_id is 4472 2024-03-29T10:46:43.688 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:43.705+0000 7f40e5ffb640 1 -- 172.21.15.88:0/3926515217 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f40f414af70 con 0x7f40f40708e0 2024-03-29T10:46:43.688 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:43.705+0000 7f40e5ffb640 1 -- 172.21.15.88:0/3926515217 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2867861120 0 0) 0x7f40d80051d0 con 0x7f40f4143d10 2024-03-29T10:46:43.688 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:43.705+0000 7f40e5ffb640 1 -- 172.21.15.88:0/3926515217 --> v1:172.21.15.196:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f40f414a310 con 0x7f40f4143d10 2024-03-29T10:46:43.688 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:43.705+0000 7f40e5ffb640 1 -- 172.21.15.88:0/3926515217 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f40f00017f0 con 0x7f40f4143d10 2024-03-29T10:46:43.688 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:43.705+0000 7f40e5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:43.688 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:43.705+0000 7f40e5ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:46:43.688 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:43.705+0000 7f40e5ffb640 10 monclient(hunting): dump: 2024-03-29T10:46:43.688 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:46:43.688 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:46:43.688 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:46:43.688 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:46:43.688 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:43.689 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:46:43.689 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:46:43.689 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:46:43.689 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:46:43.689 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:46:43.689 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:43.705+0000 7f40e5ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:46:43.689 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:43.705+0000 7f40e5ffb640 1 -- 172.21.15.88:0/3926515217 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4266772608 0 0) 0x7f40d8004eb0 con 0x7f40f40b1f60 2024-03-29T10:46:43.689 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:43.705+0000 7f40e5ffb640 1 -- 172.21.15.88:0/3926515217 --> v1:172.21.15.88:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f40d80051d0 con 0x7f40f40b1f60 2024-03-29T10:46:43.689 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:43.705+0000 7f40e5ffb640 1 -- 172.21.15.88:0/3926515217 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f40e80018a0 con 0x7f40f40b1f60 2024-03-29T10:46:43.689 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:43.705+0000 7f40e5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:43.689 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:43.705+0000 7f40e5ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:46:43.689 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:43.705+0000 7f40e5ffb640 10 monclient(hunting): dump: 2024-03-29T10:46:43.689 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:46:43.689 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:46:43.690 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:46:43.690 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:46:43.690 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:43.690 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:46:43.690 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:46:43.690 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:46:43.690 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:46:43.690 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:46:43.691 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:43.705+0000 7f40e5ffb640 1 -- 172.21.15.88:0/3926515217 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2331524899 0 0) 0x7f40f414af70 con 0x7f40f40708e0 2024-03-29T10:46:43.691 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:43.705+0000 7f40e5ffb640 1 -- 172.21.15.88:0/3926515217 --> v1:172.21.15.88:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f40d8004eb0 con 0x7f40f40708e0 2024-03-29T10:46:43.691 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:43.705+0000 7f40e5ffb640 1 -- 172.21.15.88:0/3926515217 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f40ec0018d0 con 0x7f40f40708e0 2024-03-29T10:46:43.691 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:43.705+0000 7f40e5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:43.692 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:43.705+0000 7f40e5ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:46:43.692 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:43.705+0000 7f40e5ffb640 10 monclient(hunting): dump: 2024-03-29T10:46:43.692 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:46:43.692 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:46:43.692 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:46:43.692 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:46:43.692 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:43.692 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:46:43.692 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:46:43.692 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:46:43.692 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:46:43.692 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:46:43.692 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:43.705+0000 7f40e5ffb640 1 -- 172.21.15.88:0/3926515217 <== mon.1 v1:172.21.15.196:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 929571582 0 0) 0x7f40f414a310 con 0x7f40f4143d10 2024-03-29T10:46:43.692 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:43.706+0000 7f40e5ffb640 1 -- 172.21.15.88:0/3926515217 >> v1:172.21.15.88:6790/0 conn(0x7f40f40708e0 legacy=0x7f40f41480f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:43.693 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:43.706+0000 7f40e5ffb640 1 -- 172.21.15.88:0/3926515217 >> v1:172.21.15.88:6789/0 conn(0x7f40f40b1f60 legacy=0x7f40f4148e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:43.693 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:43.706+0000 7f40e5ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:46:43.693 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:43.706+0000 7f40e5ffb640 1 monclient: found mon.b 2024-03-29T10:46:43.693 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:43.706+0000 7f40e5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:46:43.693 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:43.706+0000 7f40e5ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T10:46:43.693 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:43.706+0000 7f40e5ffb640 1 -- 172.21.15.88:0/3926515217 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40f414b1e0 con 0x7f40f4143d10 2024-03-29T10:46:43.693 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:43.706+0000 7f40e5ffb640 10 monclient: _finish_auth 0 2024-03-29T10:46:43.693 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:43.706+0000 7f40e5ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:46:43.693 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:43.706+0000 7f40e5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:46:43.693 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:43.706+0000 7f40fb5ac640 5 monclient: authenticate success, global_id 4441 2024-03-29T10:46:43.693 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:43.706+0000 7f40fb5ac640 10 monclient: _renew_subs 2024-03-29T10:46:43.693 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:43.706+0000 7f40fb5ac640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T10:46:43.693 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:43.706+0000 7f40fb5ac640 1 -- 172.21.15.88:0/3926515217 --> v1:172.21.15.196:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40f4148800 con 0x7f40f4143d10 2024-03-29T10:46:43.693 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:43.706+0000 7f40fb5ac640 10 client.4441.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:46:43.694 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:43.706+0000 7f40fb5ac640 10 monclient: _renew_subs 2024-03-29T10:46:43.694 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:43.706+0000 7f40fb5ac640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T10:46:43.694 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:43.706+0000 7f40fb5ac640 1 -- 172.21.15.88:0/3926515217 --> v1:172.21.15.196:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f40f4147970 con 0x7f40f4143d10 2024-03-29T10:46:43.694 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:43.706+0000 7f40fb5ac640 1 librados: init done 2024-03-29T10:46:43.694 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:43.707+0000 7f40e5ffb640 1 -- 172.21.15.88:0/3926515217 <== mon.1 v1:172.21.15.196:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f40f0001d00 con 0x7f40f4143d10 2024-03-29T10:46:43.694 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:43.707+0000 7f40e5ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:46:43.694 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:43.707+0000 7f40e5ffb640 1 -- 172.21.15.88:0/3926515217 <== mon.1 v1:172.21.15.196:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f40f0004aa0 con 0x7f40f4143d10 2024-03-29T10:46:43.694 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:43.707+0000 7f40e5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:43.696 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:43.707+0000 7f40e5ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:46:43.696 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:43.707+0000 7f40e5ffb640 10 monclient: dump: 2024-03-29T10:46:43.696 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:46:43.696 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:46:43.696 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:46:43.696 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:46:43.696 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:43.696 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:46:43.696 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:46:43.697 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:46:43.697 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:46:43.697 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:46:43.697 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:43.709+0000 7f40e5ffb640 1 -- 172.21.15.88:0/3926515217 <== mon.1 v1:172.21.15.196:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4235629220 0 0) 0x7f40f001aa70 con 0x7f40f4143d10 2024-03-29T10:46:43.697 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:43.709+0000 7f40e5ffb640 1 -- 172.21.15.88:0/3926515217 <== mon.1 v1:172.21.15.196:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4257673582 0 0) 0x7f40f0084a30 con 0x7f40f4143d10 2024-03-29T10:46:43.697 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:43.709+0000 7f40e5ffb640 10 client.4441.objecter ms_dispatch 0x7f40f4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:46:43.697 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:43.709+0000 7f40e5ffb640 3 client.4441.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:46:43.697 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:43.709+0000 7f40e5ffb640 3 client.4441.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:46:43.697 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:43.709+0000 7f40e5ffb640 20 client.4441.objecter dump_active .. 0 homeless 2024-03-29T10:46:43.697 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:43.709+0000 7f40e77fe640 4 set_mon_vals no callback set 2024-03-29T10:46:43.697 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:43.710+0000 7f40f9348640 10 monclient: get_auth_request con 0x7f40d806e1c0 auth_method 0 2024-03-29T10:46:43.697 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:43.710+0000 7f40e5ffb640 10 client.4441.objecter ms_handle_connect 0x7f40d806e1c0 2024-03-29T10:46:43.697 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:43.710+0000 7f40df7fe640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:46:43.697 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:43.710+0000 7f40df7fe640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:46:43.698 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:43.711+0000 7f40df7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T10:46:43.698 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:43.711+0000 7f40df7fe640 1 -- 172.21.15.88:0/3926515217 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40d4005080 con 0x7f40f4143d10 2024-03-29T10:46:43.701 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:43.718+0000 7f40e5ffb640 1 -- 172.21.15.88:0/3926515217 <== mon.1 v1:172.21.15.196:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f40f0050310 con 0x7f40f4143d10 2024-03-29T10:46:43.701 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:43.718+0000 7f40e5ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:46:43.701 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:43.718+0000 7f40e5ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:46:44.033 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:44.051+0000 7f40df7fe640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:46:44.033 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:44.051+0000 7f40df7fe640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:46:44.033 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:44.051+0000 7f40df7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T10:46:44.033 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:44.051+0000 7f40df7fe640 1 -- 172.21.15.88:0/3926515217 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f40d4005330 con 0x7f40f4143d10 2024-03-29T10:46:44.037 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:44.055+0000 7f40e5ffb640 1 -- 172.21.15.88:0/3926515217 <== mon.1 v1:172.21.15.196:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 344506374) 0x7f40f0052240 con 0x7f40f4143d10 2024-03-29T10:46:44.037 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:44.055+0000 7f40e5ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:46:44.037 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:44.055+0000 7f40e5ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:46:44.038 INFO:teuthology.misc.health.smithi088.stdout: 2024-03-29T10:46:44.038 INFO:teuthology.misc.health.smithi088.stdout:{"epoch":10,"fsid":"869f5188-f92b-42dd-a46d-8c646e0388a9","created":"2024-03-29T10:40:40.285087+0000","modified":"2024-03-29T10:40:42.536230+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:40:42.536230+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a26c0c1c-8de0-438c-a3f6-3df10ea61574","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5de24915-8dec-49b6-8b16-f3170197c591","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ac054af3-d589-4565-9644-e6af75975c53","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3ec4f045-c714-4a22-bd1a-e07330b45ab3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"cddb9867-8e30-4951-9b13-3d1e78b9c45a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dc6d9911-785d-432f-8b53-56a1ccab302f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"77c88437-b587-496e-8b4d-1824ea8b5305","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ce963d96-5a8e-44c3-99de-c0dbebe234ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:46:44.041 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:44.059+0000 7f40df7fe640 10 librados: watch_flush enter 2024-03-29T10:46:44.041 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:44.059+0000 7f40df7fe640 10 librados: watch_flush exit 2024-03-29T10:46:44.041 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:44.059+0000 7f40df7fe640 20 client.4441.objecter shutdown clearing up homeless session... 2024-03-29T10:46:44.041 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:44.059+0000 7f40df7fe640 10 client.4441.objecter successfully canceled tick 2024-03-29T10:46:44.041 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:44.059+0000 7f40df7fe640 1 -- 172.21.15.88:0/3926515217 >> v1:172.21.15.88:6816/1752861349 conn(0x7f40d806e1c0 legacy=0x7f40d8070680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:44.041 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:44.059+0000 7f40df7fe640 10 monclient: shutdown 2024-03-29T10:46:44.041 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:44.059+0000 7f40df7fe640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:46:44.042 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:44.059+0000 7f40df7fe640 1 -- 172.21.15.88:0/3926515217 >> v1:172.21.15.196:6789/0 conn(0x7f40f4143d10 legacy=0x7f40f4149c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:44.042 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:44.059+0000 7f40df7fe640 1 -- 172.21.15.88:0/3926515217 shutdown_connections 2024-03-29T10:46:44.042 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:44.059+0000 7f40df7fe640 1 -- 172.21.15.88:0/3926515217 >> 172.21.15.88:0/3926515217 conn(0x7f40f4066a00 msgr2=0x7f40f40b7d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:46:44.042 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:44.059+0000 7f40df7fe640 1 -- 172.21.15.88:0/3926515217 shutdown_connections 2024-03-29T10:46:44.042 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:44.059+0000 7f40df7fe640 1 -- 172.21.15.88:0/3926515217 wait complete. 2024-03-29T10:46:44.042 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:44.060+0000 7f40df7fe640 1 librados: shutdown 2024-03-29T10:46:44.059 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:46:50.867 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:46:51.024 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:51.039+0000 7f096359e640 10 monclient: get_monmap_and_config 2024-03-29T10:46:51.025 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:51.039+0000 7f096359e640 10 monclient: build_initial_monmap 2024-03-29T10:46:51.025 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:51.039+0000 7f096359e640 10 monclient: monmap: 2024-03-29T10:46:51.025 INFO:teuthology.misc.health.smithi088.stderr:epoch 0 2024-03-29T10:46:51.025 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:46:51.025 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:46:51.040585+0000 2024-03-29T10:46:51.025 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:46:51.040585+0000 2024-03-29T10:46:51.025 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 0 (unknown) 2024-03-29T10:46:51.025 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:46:51.025 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-03-29T10:46:51.025 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-29T10:46:51.025 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-b 2024-03-29T10:46:51.025 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:46:51.025 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:51.040+0000 7f096359e640 1 Processor -- start 2024-03-29T10:46:51.026 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:51.040+0000 7f096359e640 1 -- start start 2024-03-29T10:46:51.026 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:51.040+0000 7f096359e640 10 monclient: init 2024-03-29T10:46:51.026 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:51.041+0000 7f096359e640 10 monclient: _reopen_session rank -1 2024-03-29T10:46:51.026 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:51.041+0000 7f096359e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:46:51.026 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:51.041+0000 7f096359e640 10 monclient(hunting): picked mon.noname-a con 0x7f09640a94e0 addr v1:172.21.15.88:6789/0 2024-03-29T10:46:51.026 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:51.041+0000 7f096359e640 10 monclient(hunting): picked mon.noname-b con 0x7f09640a98c0 addr v1:172.21.15.88:6790/0 2024-03-29T10:46:51.026 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:51.041+0000 7f096359e640 10 monclient(hunting): picked mon.noname-c con 0x7f09640ae930 addr v1:172.21.15.196:6789/0 2024-03-29T10:46:51.026 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:51.041+0000 7f096359e640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09640aeda0 con 0x7f09640a94e0 2024-03-29T10:46:51.026 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:51.041+0000 7f096359e640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09640abca0 con 0x7f09640ae930 2024-03-29T10:46:51.026 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:51.041+0000 7f096359e640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09640abf80 con 0x7f09640a98c0 2024-03-29T10:46:51.026 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:51.041+0000 7f096359e640 10 monclient(hunting): _renew_subs 2024-03-29T10:46:51.026 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:51.041+0000 7f096359e640 10 monclient(hunting): authenticate will time out at 1133.277407s 2024-03-29T10:46:51.027 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:51.044+0000 7f096259c640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f09640a94e0 0x7f09640ae220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:44117/0 (socket says 172.21.15.88:44117) 2024-03-29T10:46:51.027 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:51.044+0000 7f096259c640 1 -- 172.21.15.88:0/2922694157 learned_addr learned my addr 172.21.15.88:0/2922694157 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-29T10:46:51.032 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:51.048+0000 7f096159a640 1 -- 172.21.15.88:0/2922694157 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4120654311 0 0) 0x7f09640abca0 con 0x7f09640ae930 2024-03-29T10:46:51.032 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:51.048+0000 7f096159a640 10 monclient(hunting): _negotiate 2024-03-29T10:46:51.032 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:51.048+0000 7f096159a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:51.032 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:51.048+0000 7f096159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:51.032 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:51.048+0000 7f096159a640 10 monclient(hunting): my global_id is 4444 2024-03-29T10:46:51.033 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:51.048+0000 7f096159a640 1 -- 172.21.15.88:0/2922694157 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f094c003520 con 0x7f09640ae930 2024-03-29T10:46:51.033 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:51.048+0000 7f096159a640 1 -- 172.21.15.88:0/2922694157 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3680893031 0 0) 0x7f09640aeda0 con 0x7f09640a94e0 2024-03-29T10:46:51.033 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:51.048+0000 7f096159a640 10 monclient(hunting): _negotiate 2024-03-29T10:46:51.033 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:51.048+0000 7f096159a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:51.033 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:51.048+0000 7f096159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:51.033 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:51.048+0000 7f096159a640 10 monclient(hunting): my global_id is 4446 2024-03-29T10:46:51.033 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:51.048+0000 7f096159a640 1 -- 172.21.15.88:0/2922694157 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f09640abca0 con 0x7f09640a94e0 2024-03-29T10:46:51.033 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:51.049+0000 7f096159a640 1 -- 172.21.15.88:0/2922694157 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1689041430 0 0) 0x7f09640abf80 con 0x7f09640a98c0 2024-03-29T10:46:51.033 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:51.049+0000 7f096159a640 10 monclient(hunting): _negotiate 2024-03-29T10:46:51.033 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:51.049+0000 7f096159a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:51.033 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:51.049+0000 7f096159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:51.033 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:51.049+0000 7f096159a640 10 monclient(hunting): my global_id is 4475 2024-03-29T10:46:51.033 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:51.049+0000 7f096159a640 1 -- 172.21.15.88:0/2922694157 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f09640aeda0 con 0x7f09640a98c0 2024-03-29T10:46:51.033 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:51.049+0000 7f096159a640 1 -- 172.21.15.88:0/2922694157 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3494602890 0 0) 0x7f09640abca0 con 0x7f09640a94e0 2024-03-29T10:46:51.034 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:51.049+0000 7f096159a640 1 -- 172.21.15.88:0/2922694157 >> v1:172.21.15.88:6790/0 conn(0x7f09640a98c0 legacy=0x7f09640ac3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:51.034 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:51.049+0000 7f096159a640 1 -- 172.21.15.88:0/2922694157 >> v1:172.21.15.196:6789/0 conn(0x7f09640ae930 legacy=0x7f09640acab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:51.034 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:51.049+0000 7f096159a640 10 monclient: _finish_hunting 0 2024-03-29T10:46:51.034 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:51.049+0000 7f096159a640 1 monclient: found mon.noname-a 2024-03-29T10:46:51.034 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:51.049+0000 7f096159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:46:51.034 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:51.049+0000 7f096159a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.88:6789/0 2024-03-29T10:46:51.034 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:51.049+0000 7f096159a640 1 -- 172.21.15.88:0/2922694157 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09640aadc0 con 0x7f09640a94e0 2024-03-29T10:46:51.034 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:51.049+0000 7f096159a640 10 monclient: _finish_auth 0 2024-03-29T10:46:51.034 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:51.049+0000 7f096159a640 10 monclient: _check_auth_tickets 2024-03-29T10:46:51.034 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:51.049+0000 7f096159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:46:51.035 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:51.049+0000 7f096159a640 1 -- 172.21.15.88:0/2922694157 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f09580015f0 con 0x7f09640a94e0 2024-03-29T10:46:51.035 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:51.049+0000 7f096159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:51.035 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:51.049+0000 7f096159a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:46:51.035 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:51.049+0000 7f096159a640 10 monclient: dump: 2024-03-29T10:46:51.035 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:46:51.035 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:46:51.035 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:46:51.035 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:46:51.035 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:51.035 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:46:51.035 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:46:51.035 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:46:51.035 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:46:51.036 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:46:51.036 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:51.049+0000 7f096359e640 5 monclient: authenticate success, global_id 4446 2024-03-29T10:46:51.036 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:51.049+0000 7f096359e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:46:51.036 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:51.049+0000 7f096159a640 1 -- 172.21.15.88:0/2922694157 <== mon.0 v1:172.21.15.88:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0958001810 con 0x7f09640a94e0 2024-03-29T10:46:51.036 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:51.049+0000 7f096159a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:46:51.036 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:51.049+0000 7f096159a640 1 -- 172.21.15.88:0/2922694157 <== mon.0 v1:172.21.15.88:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f0958001ba0 con 0x7f09640a94e0 2024-03-29T10:46:51.036 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:51.049+0000 7f096159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:51.036 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:51.049+0000 7f096159a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:46:51.036 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:51.049+0000 7f096159a640 10 monclient: dump: 2024-03-29T10:46:51.036 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:46:51.036 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:46:51.036 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:46:51.037 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:46:51.037 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:51.037 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:46:51.037 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:46:51.037 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:46:51.037 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:46:51.037 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:46:51.037 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:51.049+0000 7f096359e640 10 monclient: get_monmap_and_config success 2024-03-29T10:46:51.037 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:51.049+0000 7f096359e640 4 set_mon_vals no callback set 2024-03-29T10:46:51.038 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:51.050+0000 7f096359e640 10 monclient: shutdown 2024-03-29T10:46:51.038 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:51.050+0000 7f096359e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:46:51.038 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:51.050+0000 7f096359e640 1 -- 172.21.15.88:0/2922694157 >> v1:172.21.15.88:6789/0 conn(0x7f09640a94e0 legacy=0x7f09640ae220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:51.038 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:51.051+0000 7f096359e640 1 -- 172.21.15.88:0/2922694157 shutdown_connections 2024-03-29T10:46:51.039 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:51.051+0000 7f096359e640 1 -- 172.21.15.88:0/2922694157 >> 172.21.15.88:0/2922694157 conn(0x7f09640673e0 msgr2=0x7f09640b3820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:46:51.039 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:51.051+0000 7f096359e640 1 -- 172.21.15.88:0/2922694157 shutdown_connections 2024-03-29T10:46:51.039 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:51.051+0000 7f096359e640 1 -- 172.21.15.88:0/2922694157 wait complete. 2024-03-29T10:46:51.039 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:51.052+0000 7f096359e640 10 monclient: build_initial_monmap 2024-03-29T10:46:51.039 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:51.052+0000 7f096359e640 10 monclient: monmap: 2024-03-29T10:46:51.039 INFO:teuthology.misc.health.smithi088.stderr:epoch 0 2024-03-29T10:46:51.039 INFO:teuthology.misc.health.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:46:51.039 INFO:teuthology.misc.health.smithi088.stderr:last_changed 0.000000 2024-03-29T10:46:51.039 INFO:teuthology.misc.health.smithi088.stderr:created 0.000000 2024-03-29T10:46:51.039 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 0 (unknown) 2024-03-29T10:46:51.039 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:46:51.039 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-03-29T10:46:51.039 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-29T10:46:51.040 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-c 2024-03-29T10:46:51.040 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:46:51.040 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:51.052+0000 7f096359e640 1 librados: starting msgr at 2024-03-29T10:46:51.040 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:51.052+0000 7f096359e640 1 librados: starting objecter 2024-03-29T10:46:51.040 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:51.052+0000 7f096359e640 1 Processor -- start 2024-03-29T10:46:51.040 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:51.055+0000 7f096359e640 1 -- start start 2024-03-29T10:46:51.040 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:51.055+0000 7f096359e640 1 librados: setting wanted keys 2024-03-29T10:46:51.040 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:51.055+0000 7f096359e640 1 librados: calling monclient init 2024-03-29T10:46:51.040 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:51.055+0000 7f096359e640 10 monclient: init 2024-03-29T10:46:51.041 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:51.056+0000 7f096359e640 10 monclient: _reopen_session rank -1 2024-03-29T10:46:51.041 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:51.056+0000 7f096359e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:46:51.041 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:51.056+0000 7f096359e640 10 monclient(hunting): picked mon.noname-a con 0x7f09640a94e0 addr v1:172.21.15.88:6789/0 2024-03-29T10:46:51.041 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:51.056+0000 7f096359e640 10 monclient(hunting): picked mon.noname-b con 0x7f09640ae930 addr v1:172.21.15.196:6789/0 2024-03-29T10:46:51.041 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:51.056+0000 7f096359e640 10 monclient(hunting): picked mon.noname-c con 0x7f096406ea40 addr v1:172.21.15.88:6790/0 2024-03-29T10:46:51.041 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:51.056+0000 7f096359e640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0964071450 con 0x7f09640a94e0 2024-03-29T10:46:51.041 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:51.056+0000 7f096359e640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f096406f9f0 con 0x7f09640ae930 2024-03-29T10:46:51.041 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:51.056+0000 7f096359e640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0964070f90 con 0x7f096406ea40 2024-03-29T10:46:51.041 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:51.056+0000 7f096359e640 10 monclient(hunting): _renew_subs 2024-03-29T10:46:51.041 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:51.056+0000 7f096359e640 10 monclient(hunting): authenticate will time out at 1133.292739s 2024-03-29T10:46:51.043 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:51.061+0000 7f0961d9b640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f09640ae930 0x7f09640718e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.88:41041/0 (socket says 172.21.15.88:41041) 2024-03-29T10:46:51.043 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:51.061+0000 7f0961d9b640 1 -- 172.21.15.88:0/1382752052 learned_addr learned my addr 172.21.15.88:0/1382752052 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-29T10:46:51.046 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:51.063+0000 7f0952ffd640 10 client.?.objecter ms_handle_connect 0x7f09640ae930 2024-03-29T10:46:51.046 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:51.063+0000 7f0952ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:46:51.049 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:51.067+0000 7f0952ffd640 10 client.?.objecter ms_handle_connect 0x7f096406ea40 2024-03-29T10:46:51.049 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:51.067+0000 7f0952ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:46:51.050 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:51.067+0000 7f0952ffd640 1 -- 172.21.15.88:0/1382752052 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1322275174 0 0) 0x7f096406f9f0 con 0x7f09640ae930 2024-03-29T10:46:51.050 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:51.067+0000 7f0952ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:46:51.050 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:51.067+0000 7f0952ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:51.050 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:51.067+0000 7f0952ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:51.050 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:51.067+0000 7f0952ffd640 10 monclient(hunting): my global_id is 4447 2024-03-29T10:46:51.050 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:51.067+0000 7f0952ffd640 1 -- 172.21.15.88:0/1382752052 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0944004eb0 con 0x7f09640ae930 2024-03-29T10:46:51.050 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:51.067+0000 7f0952ffd640 10 client.?.objecter ms_handle_connect 0x7f09640a94e0 2024-03-29T10:46:51.050 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:51.067+0000 7f0952ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:46:51.051 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:51.067+0000 7f0952ffd640 1 -- 172.21.15.88:0/1382752052 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3137063199 0 0) 0x7f0944004eb0 con 0x7f09640ae930 2024-03-29T10:46:51.051 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:51.067+0000 7f0952ffd640 1 -- 172.21.15.88:0/1382752052 --> v1:172.21.15.196:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f096406f9f0 con 0x7f09640ae930 2024-03-29T10:46:51.051 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:51.067+0000 7f0952ffd640 1 -- 172.21.15.88:0/1382752052 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f0954001a30 con 0x7f09640ae930 2024-03-29T10:46:51.051 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:51.067+0000 7f0952ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:51.051 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:51.067+0000 7f0952ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:46:51.051 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:51.067+0000 7f0952ffd640 10 monclient(hunting): dump: 2024-03-29T10:46:51.051 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:46:51.051 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:46:51.051 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:46:51.051 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:46:51.051 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:51.051 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:46:51.051 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:46:51.052 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:46:51.052 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:46:51.052 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:46:51.052 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:51.067+0000 7f0952ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:46:51.053 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:51.069+0000 7f0952ffd640 1 -- 172.21.15.88:0/1382752052 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2741031737 0 0) 0x7f0964070f90 con 0x7f096406ea40 2024-03-29T10:46:51.053 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:51.069+0000 7f0952ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:46:51.053 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:51.069+0000 7f0952ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:51.053 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:51.069+0000 7f0952ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:51.053 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:51.069+0000 7f0952ffd640 10 monclient(hunting): my global_id is 4478 2024-03-29T10:46:51.053 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:51.069+0000 7f0952ffd640 1 -- 172.21.15.88:0/1382752052 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0944004eb0 con 0x7f096406ea40 2024-03-29T10:46:51.053 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:51.070+0000 7f0952ffd640 1 -- 172.21.15.88:0/1382752052 <== mon.1 v1:172.21.15.196:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 712184642 0 0) 0x7f096406f9f0 con 0x7f09640ae930 2024-03-29T10:46:51.053 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:51.070+0000 7f0952ffd640 1 -- 172.21.15.88:0/1382752052 >> v1:172.21.15.88:6790/0 conn(0x7f096406ea40 legacy=0x7f0964071ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:51.054 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:51.070+0000 7f0952ffd640 1 -- 172.21.15.88:0/1382752052 >> v1:172.21.15.88:6789/0 conn(0x7f09640a94e0 legacy=0x7f096406f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:51.054 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:51.070+0000 7f0952ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:46:51.054 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:51.070+0000 7f0952ffd640 1 monclient: found mon.b 2024-03-29T10:46:51.054 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:51.070+0000 7f0952ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:46:51.054 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:51.070+0000 7f0952ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T10:46:51.054 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:51.070+0000 7f0952ffd640 1 -- 172.21.15.88:0/1382752052 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0964071120 con 0x7f09640ae930 2024-03-29T10:46:51.054 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:51.070+0000 7f0952ffd640 10 monclient: _finish_auth 0 2024-03-29T10:46:51.054 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:51.070+0000 7f0952ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:46:51.054 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:51.070+0000 7f0952ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:46:51.054 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:51.070+0000 7f096359e640 5 monclient: authenticate success, global_id 4447 2024-03-29T10:46:51.054 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:51.070+0000 7f096359e640 10 monclient: _renew_subs 2024-03-29T10:46:51.054 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:51.070+0000 7f096359e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T10:46:51.054 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:51.070+0000 7f096359e640 1 -- 172.21.15.88:0/1382752052 --> v1:172.21.15.196:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0964200680 con 0x7f09640ae930 2024-03-29T10:46:51.055 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:51.070+0000 7f096359e640 10 client.4447.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:46:51.055 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:51.070+0000 7f096359e640 10 monclient: _renew_subs 2024-03-29T10:46:51.055 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:51.070+0000 7f096359e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T10:46:51.055 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:51.070+0000 7f096359e640 1 -- 172.21.15.88:0/1382752052 --> v1:172.21.15.196:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0964200a40 con 0x7f09640ae930 2024-03-29T10:46:51.055 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:51.070+0000 7f096359e640 1 librados: init done 2024-03-29T10:46:51.055 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:51.070+0000 7f0952ffd640 1 -- 172.21.15.88:0/1382752052 <== mon.1 v1:172.21.15.196:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0954001cb0 con 0x7f09640ae930 2024-03-29T10:46:51.055 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:51.070+0000 7f0952ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:46:51.055 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:51.070+0000 7f0952ffd640 1 -- 172.21.15.88:0/1382752052 <== mon.1 v1:172.21.15.196:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f0954003a60 con 0x7f09640ae930 2024-03-29T10:46:51.055 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:51.070+0000 7f0952ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:51.055 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:51.070+0000 7f0952ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:46:51.055 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:51.070+0000 7f0952ffd640 10 monclient: dump: 2024-03-29T10:46:51.055 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:46:51.055 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:46:51.055 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:46:51.055 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:46:51.056 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:51.056 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:46:51.056 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:46:51.056 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:46:51.056 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:46:51.056 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:46:51.056 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:51.070+0000 7f0960d99640 4 set_mon_vals no callback set 2024-03-29T10:46:51.057 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:51.072+0000 7f0952ffd640 1 -- 172.21.15.88:0/1382752052 <== mon.1 v1:172.21.15.196:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4235629220 0 0) 0x7f0954003bc0 con 0x7f09640ae930 2024-03-29T10:46:51.057 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:51.073+0000 7f0952ffd640 1 -- 172.21.15.88:0/1382752052 <== mon.1 v1:172.21.15.196:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4257673582 0 0) 0x7f0954084b10 con 0x7f09640ae930 2024-03-29T10:46:51.057 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:51.073+0000 7f0952ffd640 10 client.4447.objecter ms_dispatch 0x7f0964000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:46:51.057 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:51.073+0000 7f0952ffd640 3 client.4447.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:46:51.058 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:51.073+0000 7f0952ffd640 3 client.4447.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:46:51.058 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:51.073+0000 7f0952ffd640 20 client.4447.objecter dump_active .. 0 homeless 2024-03-29T10:46:51.058 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:51.073+0000 7f096259c640 10 monclient: get_auth_request con 0x7f094406dd60 auth_method 0 2024-03-29T10:46:51.058 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:51.073+0000 7f0952ffd640 10 client.4447.objecter ms_handle_connect 0x7f094406dd60 2024-03-29T10:46:51.058 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:51.074+0000 7f096359e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:46:51.058 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:51.074+0000 7f096359e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:46:51.058 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:51.074+0000 7f096359e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T10:46:51.058 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:51.074+0000 7f096359e640 1 -- 172.21.15.88:0/1382752052 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0948005080 con 0x7f09640ae930 2024-03-29T10:46:51.065 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:51.083+0000 7f0952ffd640 1 -- 172.21.15.88:0/1382752052 <== mon.1 v1:172.21.15.196:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f09540503f0 con 0x7f09640ae930 2024-03-29T10:46:51.065 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:51.083+0000 7f0952ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:46:51.065 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:51.083+0000 7f0952ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:46:51.394 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:51.411+0000 7f096359e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:46:51.394 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:51.411+0000 7f096359e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:46:51.395 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:51.411+0000 7f096359e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T10:46:51.395 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:51.411+0000 7f096359e640 1 -- 172.21.15.88:0/1382752052 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0948005330 con 0x7f09640ae930 2024-03-29T10:46:51.395 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:51.413+0000 7f0952ffd640 1 -- 172.21.15.88:0/1382752052 <== mon.1 v1:172.21.15.196:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 344506374) 0x7f0954052320 con 0x7f09640ae930 2024-03-29T10:46:51.395 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:51.413+0000 7f0952ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:46:51.395 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:51.413+0000 7f0952ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:46:51.396 INFO:teuthology.misc.health.smithi088.stdout: 2024-03-29T10:46:51.397 INFO:teuthology.misc.health.smithi088.stdout:{"epoch":10,"fsid":"869f5188-f92b-42dd-a46d-8c646e0388a9","created":"2024-03-29T10:40:40.285087+0000","modified":"2024-03-29T10:40:42.536230+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:40:42.536230+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a26c0c1c-8de0-438c-a3f6-3df10ea61574","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5de24915-8dec-49b6-8b16-f3170197c591","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ac054af3-d589-4565-9644-e6af75975c53","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3ec4f045-c714-4a22-bd1a-e07330b45ab3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"cddb9867-8e30-4951-9b13-3d1e78b9c45a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dc6d9911-785d-432f-8b53-56a1ccab302f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"77c88437-b587-496e-8b4d-1824ea8b5305","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ce963d96-5a8e-44c3-99de-c0dbebe234ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:46:51.400 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:51.417+0000 7f096359e640 10 librados: watch_flush enter 2024-03-29T10:46:51.400 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:51.418+0000 7f096359e640 10 librados: watch_flush exit 2024-03-29T10:46:51.401 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:51.418+0000 7f096359e640 20 client.4447.objecter shutdown clearing up homeless session... 2024-03-29T10:46:51.401 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:51.418+0000 7f096359e640 10 client.4447.objecter successfully canceled tick 2024-03-29T10:46:51.401 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:51.418+0000 7f096359e640 1 -- 172.21.15.88:0/1382752052 >> v1:172.21.15.88:6816/1752861349 conn(0x7f094406dd60 legacy=0x7f0944070220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:51.401 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:51.418+0000 7f096359e640 10 monclient: shutdown 2024-03-29T10:46:51.401 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:51.418+0000 7f096359e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:46:51.401 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:51.418+0000 7f096359e640 1 -- 172.21.15.88:0/1382752052 >> v1:172.21.15.196:6789/0 conn(0x7f09640ae930 legacy=0x7f09640718e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:51.401 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:51.418+0000 7f096359e640 1 -- 172.21.15.88:0/1382752052 shutdown_connections 2024-03-29T10:46:51.401 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:51.418+0000 7f096359e640 1 -- 172.21.15.88:0/1382752052 >> 172.21.15.88:0/1382752052 conn(0x7f09640673e0 msgr2=0x7f0964154b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:46:51.401 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:51.418+0000 7f096359e640 1 -- 172.21.15.88:0/1382752052 shutdown_connections 2024-03-29T10:46:51.401 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:51.418+0000 7f096359e640 1 -- 172.21.15.88:0/1382752052 wait complete. 2024-03-29T10:46:51.402 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:51.419+0000 7f096359e640 1 librados: shutdown 2024-03-29T10:46:51.421 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:46:58.228 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:46:58.379 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:58.397+0000 7f2617233640 10 monclient: get_monmap_and_config 2024-03-29T10:46:58.380 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:58.397+0000 7f2617233640 10 monclient: build_initial_monmap 2024-03-29T10:46:58.380 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:58.397+0000 7f2617233640 10 monclient: monmap: 2024-03-29T10:46:58.380 INFO:teuthology.misc.health.smithi088.stderr:epoch 0 2024-03-29T10:46:58.380 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:46:58.380 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:46:58.398450+0000 2024-03-29T10:46:58.380 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:46:58.398450+0000 2024-03-29T10:46:58.380 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 0 (unknown) 2024-03-29T10:46:58.380 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:46:58.380 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-03-29T10:46:58.380 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-29T10:46:58.380 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-b 2024-03-29T10:46:58.380 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:46:58.381 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:58.398+0000 7f2617233640 1 Processor -- start 2024-03-29T10:46:58.381 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:58.398+0000 7f2617233640 1 -- start start 2024-03-29T10:46:58.381 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:58.398+0000 7f2617233640 10 monclient: init 2024-03-29T10:46:58.387 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:58.399+0000 7f2617233640 10 monclient: _reopen_session rank -1 2024-03-29T10:46:58.387 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:58.399+0000 7f2617233640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:46:58.387 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:58.399+0000 7f2617233640 10 monclient(hunting): picked mon.noname-c con 0x7f26100b1f60 addr v1:172.21.15.196:6789/0 2024-03-29T10:46:58.387 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:58.399+0000 7f2617233640 10 monclient(hunting): picked mon.noname-b con 0x7f2610071eb0 addr v1:172.21.15.88:6790/0 2024-03-29T10:46:58.388 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:58.399+0000 7f2617233640 10 monclient(hunting): picked mon.noname-a con 0x7f2610143d10 addr v1:172.21.15.88:6789/0 2024-03-29T10:46:58.388 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:58.399+0000 7f2617233640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f261006ef80 con 0x7f2610143d10 2024-03-29T10:46:58.388 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:58.399+0000 7f2617233640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f261006f260 con 0x7f26100b1f60 2024-03-29T10:46:58.388 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:58.399+0000 7f2617233640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26100b40c0 con 0x7f2610071eb0 2024-03-29T10:46:58.388 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:58.399+0000 7f2617233640 10 monclient(hunting): _renew_subs 2024-03-29T10:46:58.388 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:58.399+0000 7f2617233640 10 monclient(hunting): authenticate will time out at 1140.635404s 2024-03-29T10:46:58.388 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:58.406+0000 7f260ffff640 1 --1- >> v1:172.21.15.88:6790/0 conn(0x7f2610071eb0 0x7f2610146470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6790/0 says I am v1:172.21.15.88:40369/0 (socket says 172.21.15.88:40369) 2024-03-29T10:46:58.389 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:58.406+0000 7f260ffff640 1 -- 172.21.15.88:0/3545745574 learned_addr learned my addr 172.21.15.88:0/3545745574 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-29T10:46:58.389 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:58.406+0000 7f260f7fe640 1 -- 172.21.15.88:0/3545745574 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2548454165 0 0) 0x7f26100b40c0 con 0x7f2610071eb0 2024-03-29T10:46:58.389 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:58.406+0000 7f260f7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:46:58.389 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:58.406+0000 7f260f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:58.389 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:58.406+0000 7f260f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:58.389 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:58.406+0000 7f260f7fe640 10 monclient(hunting): my global_id is 4481 2024-03-29T10:46:58.389 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:58.406+0000 7f260f7fe640 1 -- 172.21.15.88:0/3545745574 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f25f8003520 con 0x7f2610071eb0 2024-03-29T10:46:58.389 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:58.406+0000 7f260f7fe640 1 -- 172.21.15.88:0/3545745574 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4090394873 0 0) 0x7f25f8003520 con 0x7f2610071eb0 2024-03-29T10:46:58.389 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:58.406+0000 7f260f7fe640 1 -- 172.21.15.88:0/3545745574 >> v1:172.21.15.196:6789/0 conn(0x7f26100b1f60 legacy=0x7f2610143600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:46:58.389 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:58.406+0000 7f260f7fe640 1 -- 172.21.15.88:0/3545745574 >> v1:172.21.15.88:6789/0 conn(0x7f2610143d10 legacy=0x7f2610146b80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:46:58.393 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:58.410+0000 7f260f7fe640 10 monclient: _finish_hunting 0 2024-03-29T10:46:58.393 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:58.410+0000 7f260f7fe640 1 monclient: found mon.noname-b 2024-03-29T10:46:58.393 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:58.410+0000 7f260f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:46:58.393 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:58.410+0000 7f260f7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.88:6790/0 2024-03-29T10:46:58.393 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:58.410+0000 7f260f7fe640 1 -- 172.21.15.88:0/3545745574 --> v1:172.21.15.88:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2610155d70 con 0x7f2610071eb0 2024-03-29T10:46:58.394 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:58.410+0000 7f260f7fe640 10 monclient: _finish_auth 0 2024-03-29T10:46:58.394 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:58.410+0000 7f260f7fe640 10 monclient: _check_auth_tickets 2024-03-29T10:46:58.394 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:58.410+0000 7f260f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:46:58.394 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:58.410+0000 7f2617233640 5 monclient: authenticate success, global_id 4481 2024-03-29T10:46:58.394 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:58.410+0000 7f2617233640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:46:58.394 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:58.410+0000 7f260f7fe640 1 -- 172.21.15.88:0/3545745574 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f2608001a10 con 0x7f2610071eb0 2024-03-29T10:46:58.394 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:58.410+0000 7f260f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:58.394 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:58.410+0000 7f260f7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:46:58.394 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:58.410+0000 7f260f7fe640 10 monclient: dump: 2024-03-29T10:46:58.394 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:46:58.394 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:46:58.394 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:46:58.394 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:46:58.394 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:58.395 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:46:58.395 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:46:58.395 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:46:58.395 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:46:58.395 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:46:58.395 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:58.410+0000 7f2617233640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:46:58.396 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:58.412+0000 7f260f7fe640 1 -- 172.21.15.88:0/3545745574 <== mon.2 v1:172.21.15.88:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2608001be0 con 0x7f2610071eb0 2024-03-29T10:46:58.396 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:58.412+0000 7f260f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:46:58.396 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:58.412+0000 7f260f7fe640 1 -- 172.21.15.88:0/3545745574 <== mon.2 v1:172.21.15.88:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f2608002030 con 0x7f2610071eb0 2024-03-29T10:46:58.396 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:58.412+0000 7f260f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:58.396 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:58.412+0000 7f260f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:46:58.396 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:58.412+0000 7f260f7fe640 10 monclient: dump: 2024-03-29T10:46:58.396 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:46:58.396 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:46:58.397 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:46:58.397 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:46:58.397 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:58.397 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:46:58.397 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:46:58.397 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:46:58.397 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:46:58.397 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:46:58.397 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:58.412+0000 7f2617233640 10 monclient: get_monmap_and_config success 2024-03-29T10:46:58.397 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:58.412+0000 7f2617233640 4 set_mon_vals no callback set 2024-03-29T10:46:58.397 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:58.413+0000 7f2617233640 10 monclient: shutdown 2024-03-29T10:46:58.397 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:58.413+0000 7f2617233640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:46:58.397 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:58.413+0000 7f2617233640 1 -- 172.21.15.88:0/3545745574 >> v1:172.21.15.88:6790/0 conn(0x7f2610071eb0 legacy=0x7f2610146470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:58.397 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:58.413+0000 7f2617233640 1 -- 172.21.15.88:0/3545745574 shutdown_connections 2024-03-29T10:46:58.397 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:58.413+0000 7f2617233640 1 -- 172.21.15.88:0/3545745574 >> 172.21.15.88:0/3545745574 conn(0x7f2610066a00 msgr2=0x7f2610066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:46:58.398 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:58.413+0000 7f2617233640 1 -- 172.21.15.88:0/3545745574 shutdown_connections 2024-03-29T10:46:58.399 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:58.413+0000 7f2617233640 1 -- 172.21.15.88:0/3545745574 wait complete. 2024-03-29T10:46:58.399 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:58.414+0000 7f2617233640 10 monclient: build_initial_monmap 2024-03-29T10:46:58.399 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:58.414+0000 7f2617233640 10 monclient: monmap: 2024-03-29T10:46:58.399 INFO:teuthology.misc.health.smithi088.stderr:epoch 0 2024-03-29T10:46:58.399 INFO:teuthology.misc.health.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:46:58.399 INFO:teuthology.misc.health.smithi088.stderr:last_changed 0.000000 2024-03-29T10:46:58.400 INFO:teuthology.misc.health.smithi088.stderr:created 0.000000 2024-03-29T10:46:58.400 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 0 (unknown) 2024-03-29T10:46:58.400 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:46:58.400 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-03-29T10:46:58.400 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-29T10:46:58.400 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-c 2024-03-29T10:46:58.400 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:46:58.400 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:58.414+0000 7f2617233640 1 librados: starting msgr at 2024-03-29T10:46:58.405 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:58.414+0000 7f2617233640 1 librados: starting objecter 2024-03-29T10:46:58.405 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:58.414+0000 7f2617233640 1 Processor -- start 2024-03-29T10:46:58.405 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:58.415+0000 7f2617233640 1 -- start start 2024-03-29T10:46:58.405 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:58.415+0000 7f2617233640 1 librados: setting wanted keys 2024-03-29T10:46:58.405 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:58.415+0000 7f2617233640 1 librados: calling monclient init 2024-03-29T10:46:58.405 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:58.415+0000 7f2617233640 10 monclient: init 2024-03-29T10:46:58.405 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:58.416+0000 7f2617233640 10 monclient: _reopen_session rank -1 2024-03-29T10:46:58.405 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:58.416+0000 7f2617233640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:46:58.405 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:58.416+0000 7f2617233640 10 monclient(hunting): picked mon.noname-c con 0x7f26100b1f60 addr v1:172.21.15.88:6790/0 2024-03-29T10:46:58.406 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:58.416+0000 7f2617233640 10 monclient(hunting): picked mon.noname-a con 0x7f2610143d10 addr v1:172.21.15.88:6789/0 2024-03-29T10:46:58.406 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:58.416+0000 7f2617233640 10 monclient(hunting): picked mon.noname-b con 0x7f261006f620 addr v1:172.21.15.196:6789/0 2024-03-29T10:46:58.406 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:58.416+0000 7f2617233640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26101328d0 con 0x7f2610143d10 2024-03-29T10:46:58.406 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:58.416+0000 7f2617233640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26100703a0 con 0x7f261006f620 2024-03-29T10:46:58.406 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:58.416+0000 7f2617233640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2610070680 con 0x7f26100b1f60 2024-03-29T10:46:58.406 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:58.416+0000 7f2617233640 10 monclient(hunting): _renew_subs 2024-03-29T10:46:58.406 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:58.416+0000 7f2617233640 10 monclient(hunting): authenticate will time out at 1140.652184s 2024-03-29T10:46:58.406 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:58.420+0000 7f260ffff640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f2610143d10 0x7f2610131ab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:58205/0 (socket says 172.21.15.88:58205) 2024-03-29T10:46:58.406 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:58.420+0000 7f260ffff640 1 -- 172.21.15.88:0/1137743716 learned_addr learned my addr 172.21.15.88:0/1137743716 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-29T10:46:58.406 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:58.420+0000 7f260d7fa640 10 client.?.objecter ms_handle_connect 0x7f26100b1f60 2024-03-29T10:46:58.406 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:58.420+0000 7f260d7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:46:58.406 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:58.421+0000 7f260d7fa640 1 -- 172.21.15.88:0/1137743716 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 734467566 0 0) 0x7f2610070680 con 0x7f26100b1f60 2024-03-29T10:46:58.407 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:58.421+0000 7f260d7fa640 10 monclient(hunting): _negotiate 2024-03-29T10:46:58.407 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:58.421+0000 7f260d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:58.407 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:58.421+0000 7f260d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:58.407 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:58.421+0000 7f260d7fa640 10 monclient(hunting): my global_id is 4484 2024-03-29T10:46:58.407 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:58.421+0000 7f260d7fa640 1 -- 172.21.15.88:0/1137743716 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f25fc004eb0 con 0x7f26100b1f60 2024-03-29T10:46:58.407 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:58.421+0000 7f260d7fa640 1 -- 172.21.15.88:0/1137743716 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2022731030 0 0) 0x7f25fc004eb0 con 0x7f26100b1f60 2024-03-29T10:46:58.407 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:58.421+0000 7f260d7fa640 1 -- 172.21.15.88:0/1137743716 --> v1:172.21.15.88:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f25fc005300 con 0x7f26100b1f60 2024-03-29T10:46:58.407 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:58.421+0000 7f260d7fa640 1 -- 172.21.15.88:0/1137743716 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f2604001830 con 0x7f26100b1f60 2024-03-29T10:46:58.407 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:58.421+0000 7f260d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:58.407 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:58.421+0000 7f260d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:46:58.407 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:58.421+0000 7f260d7fa640 10 monclient(hunting): dump: 2024-03-29T10:46:58.407 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:46:58.407 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:46:58.408 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:46:58.408 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:46:58.408 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:58.408 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:46:58.408 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:46:58.408 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:46:58.408 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:46:58.408 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:46:58.408 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:58.421+0000 7f260d7fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:46:58.408 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:58.422+0000 7f260d7fa640 1 -- 172.21.15.88:0/1137743716 <== mon.2 v1:172.21.15.88:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1419494542 0 0) 0x7f25fc005300 con 0x7f26100b1f60 2024-03-29T10:46:58.408 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:58.422+0000 7f260d7fa640 1 -- 172.21.15.88:0/1137743716 >> v1:172.21.15.196:6789/0 conn(0x7f261006f620 legacy=0x7f26101321c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:46:58.411 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:58.423+0000 7f260d7fa640 1 -- 172.21.15.88:0/1137743716 >> v1:172.21.15.88:6789/0 conn(0x7f2610143d10 legacy=0x7f2610131ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:58.411 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:58.423+0000 7f260d7fa640 10 monclient: _finish_hunting 0 2024-03-29T10:46:58.411 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:58.423+0000 7f260d7fa640 1 monclient: found mon.c 2024-03-29T10:46:58.411 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:58.423+0000 7f260d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:46:58.411 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:58.423+0000 7f260d7fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-03-29T10:46:58.412 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:58.423+0000 7f260d7fa640 1 -- 172.21.15.88:0/1137743716 --> v1:172.21.15.88:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2610070960 con 0x7f26100b1f60 2024-03-29T10:46:58.412 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:58.423+0000 7f260d7fa640 10 monclient: _finish_auth 0 2024-03-29T10:46:58.412 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:58.423+0000 7f260d7fa640 10 monclient: _check_auth_tickets 2024-03-29T10:46:58.412 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:58.423+0000 7f260d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:46:58.412 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:58.423+0000 7f260d7fa640 10 client.?.objecter ms_handle_connect 0x7f2610143d10 2024-03-29T10:46:58.412 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:58.423+0000 7f260d7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:46:58.412 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:58.424+0000 7f2617233640 5 monclient: authenticate success, global_id 4484 2024-03-29T10:46:58.412 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:58.424+0000 7f2617233640 10 monclient: _renew_subs 2024-03-29T10:46:58.412 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:58.424+0000 7f2617233640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-03-29T10:46:58.412 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:58.424+0000 7f2617233640 1 -- 172.21.15.88:0/1137743716 --> v1:172.21.15.88:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26101544d0 con 0x7f26100b1f60 2024-03-29T10:46:58.412 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:58.424+0000 7f2617233640 10 client.4484.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:46:58.412 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:58.424+0000 7f2617233640 10 monclient: _renew_subs 2024-03-29T10:46:58.412 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:58.424+0000 7f2617233640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-03-29T10:46:58.412 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:58.424+0000 7f2617233640 1 -- 172.21.15.88:0/1137743716 --> v1:172.21.15.88:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2610155220 con 0x7f26100b1f60 2024-03-29T10:46:58.412 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:58.424+0000 7f2617233640 1 librados: init done 2024-03-29T10:46:58.413 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:58.426+0000 7f2617233640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:46:58.413 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:58.426+0000 7f2617233640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:46:58.413 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:58.426+0000 7f2617233640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-03-29T10:46:58.413 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:58.426+0000 7f2617233640 1 -- 172.21.15.88:0/1137743716 --> v1:172.21.15.88:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2610075f10 con 0x7f26100b1f60 2024-03-29T10:46:58.413 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:58.428+0000 7f260d7fa640 1 -- 172.21.15.88:0/1137743716 <== mon.2 v1:172.21.15.88:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f26040035f0 con 0x7f26100b1f60 2024-03-29T10:46:58.413 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:58.428+0000 7f260d7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:46:58.413 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:58.428+0000 7f260effd640 4 set_mon_vals no callback set 2024-03-29T10:46:58.414 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:58.430+0000 7f260d7fa640 1 -- 172.21.15.88:0/1137743716 <== mon.2 v1:172.21.15.88:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f2604001830 con 0x7f26100b1f60 2024-03-29T10:46:58.415 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:58.430+0000 7f260d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:58.415 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:58.430+0000 7f260d7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:46:58.416 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:58.430+0000 7f260d7fa640 10 monclient: dump: 2024-03-29T10:46:58.416 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:46:58.416 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:46:58.416 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:46:58.416 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:46:58.416 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:58.416 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:46:58.416 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:46:58.416 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:46:58.416 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:46:58.416 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:46:58.416 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:58.430+0000 7f260d7fa640 1 -- 172.21.15.88:0/1137743716 <== mon.2 v1:172.21.15.88:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4235629220 0 0) 0x7f2604019650 con 0x7f26100b1f60 2024-03-29T10:46:58.416 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:58.431+0000 7f260d7fa640 1 -- 172.21.15.88:0/1137743716 <== mon.2 v1:172.21.15.88:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4257673582 0 0) 0x7f2604083670 con 0x7f26100b1f60 2024-03-29T10:46:58.416 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:58.431+0000 7f260d7fa640 10 client.4484.objecter ms_dispatch 0x7f2610000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:46:58.417 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:58.431+0000 7f260d7fa640 3 client.4484.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:46:58.417 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:58.431+0000 7f260d7fa640 3 client.4484.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:46:58.417 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:58.431+0000 7f260d7fa640 20 client.4484.objecter dump_active .. 0 homeless 2024-03-29T10:46:58.417 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:58.431+0000 7f260ffff640 10 monclient: get_auth_request con 0x7f25fc06e070 auth_method 0 2024-03-29T10:46:58.417 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:58.433+0000 7f260d7fa640 10 client.4484.objecter ms_handle_connect 0x7f25fc06e070 2024-03-29T10:46:58.420 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:58.438+0000 7f260d7fa640 1 -- 172.21.15.88:0/1137743716 <== mon.2 v1:172.21.15.88:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f260404ee00 con 0x7f26100b1f60 2024-03-29T10:46:58.420 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:58.438+0000 7f260d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:46:58.421 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:58.438+0000 7f260d7fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:46:58.763 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:58.780+0000 7f2617233640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:46:58.763 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:58.780+0000 7f2617233640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:46:58.763 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:58.780+0000 7f2617233640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-03-29T10:46:58.763 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:58.780+0000 7f2617233640 1 -- 172.21.15.88:0/1137743716 --> v1:172.21.15.88:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2610074f50 con 0x7f26100b1f60 2024-03-29T10:46:58.764 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:58.781+0000 7f260d7fa640 1 -- 172.21.15.88:0/1137743716 <== mon.2 v1:172.21.15.88:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 344506374) 0x7f2604050f00 con 0x7f26100b1f60 2024-03-29T10:46:58.764 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:58.781+0000 7f260d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:46:58.764 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:58.781+0000 7f260d7fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:46:58.764 INFO:teuthology.misc.health.smithi088.stdout: 2024-03-29T10:46:58.764 INFO:teuthology.misc.health.smithi088.stdout:{"epoch":10,"fsid":"869f5188-f92b-42dd-a46d-8c646e0388a9","created":"2024-03-29T10:40:40.285087+0000","modified":"2024-03-29T10:40:42.536230+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:40:42.536230+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a26c0c1c-8de0-438c-a3f6-3df10ea61574","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5de24915-8dec-49b6-8b16-f3170197c591","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ac054af3-d589-4565-9644-e6af75975c53","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3ec4f045-c714-4a22-bd1a-e07330b45ab3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"cddb9867-8e30-4951-9b13-3d1e78b9c45a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dc6d9911-785d-432f-8b53-56a1ccab302f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"77c88437-b587-496e-8b4d-1824ea8b5305","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ce963d96-5a8e-44c3-99de-c0dbebe234ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:46:58.767 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:58.785+0000 7f2617233640 10 librados: watch_flush enter 2024-03-29T10:46:58.768 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:58.785+0000 7f2617233640 10 librados: watch_flush exit 2024-03-29T10:46:58.768 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:58.785+0000 7f2617233640 20 client.4484.objecter shutdown clearing up homeless session... 2024-03-29T10:46:58.768 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:58.785+0000 7f2617233640 10 client.4484.objecter successfully canceled tick 2024-03-29T10:46:58.768 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:58.785+0000 7f2617233640 1 -- 172.21.15.88:0/1137743716 >> v1:172.21.15.88:6816/1752861349 conn(0x7f25fc06e070 legacy=0x7f25fc070530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:58.768 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:58.785+0000 7f2617233640 10 monclient: shutdown 2024-03-29T10:46:58.768 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:58.785+0000 7f2617233640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:46:58.768 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:58.785+0000 7f2617233640 1 -- 172.21.15.88:0/1137743716 >> v1:172.21.15.88:6790/0 conn(0x7f26100b1f60 legacy=0x7f2610071010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:58.768 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:58.785+0000 7f2617233640 1 -- 172.21.15.88:0/1137743716 shutdown_connections 2024-03-29T10:46:58.768 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:58.785+0000 7f2617233640 1 -- 172.21.15.88:0/1137743716 >> 172.21.15.88:0/1137743716 conn(0x7f2610066a00 msgr2=0x7f26100b5810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:46:58.768 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:58.785+0000 7f2617233640 1 -- 172.21.15.88:0/1137743716 shutdown_connections 2024-03-29T10:46:58.769 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:58.786+0000 7f2617233640 1 -- 172.21.15.88:0/1137743716 wait complete. 2024-03-29T10:46:58.769 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:46:58.786+0000 7f2617233640 1 librados: shutdown 2024-03-29T10:46:58.783 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:47:05.591 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:47:05.741 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:05.760+0000 7f01a489a640 10 monclient: get_monmap_and_config 2024-03-29T10:47:05.741 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:05.760+0000 7f01a489a640 10 monclient: build_initial_monmap 2024-03-29T10:47:05.741 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:05.760+0000 7f01a489a640 10 monclient: monmap: 2024-03-29T10:47:05.742 INFO:teuthology.misc.health.smithi088.stderr:epoch 0 2024-03-29T10:47:05.742 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:47:05.742 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:47:05.760121+0000 2024-03-29T10:47:05.742 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:47:05.760121+0000 2024-03-29T10:47:05.742 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 0 (unknown) 2024-03-29T10:47:05.742 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:47:05.742 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-03-29T10:47:05.742 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-29T10:47:05.742 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-b 2024-03-29T10:47:05.742 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:47:05.743 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:05.761+0000 7f01a489a640 1 Processor -- start 2024-03-29T10:47:05.743 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:05.761+0000 7f01a489a640 1 -- start start 2024-03-29T10:47:05.743 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:05.761+0000 7f01a489a640 10 monclient: init 2024-03-29T10:47:05.743 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:05.762+0000 7f01a489a640 10 monclient: _reopen_session rank -1 2024-03-29T10:47:05.747 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:05.762+0000 7f01a489a640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:47:05.747 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:05.762+0000 7f01a489a640 10 monclient(hunting): picked mon.noname-a con 0x7f01a00b0500 addr v1:172.21.15.88:6789/0 2024-03-29T10:47:05.747 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:05.762+0000 7f01a489a640 10 monclient(hunting): picked mon.noname-b con 0x7f01a00b08e0 addr v1:172.21.15.88:6790/0 2024-03-29T10:47:05.747 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:05.765+0000 7f01a489a640 10 monclient(hunting): picked mon.noname-c con 0x7f01a00aec70 addr v1:172.21.15.196:6789/0 2024-03-29T10:47:05.747 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:05.765+0000 7f01a489a640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01a00ae7e0 con 0x7f01a00b0500 2024-03-29T10:47:05.748 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:05.765+0000 7f01a489a640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01a013daf0 con 0x7f01a00aec70 2024-03-29T10:47:05.748 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:05.765+0000 7f01a489a640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01a00accf0 con 0x7f01a00b08e0 2024-03-29T10:47:05.748 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:05.765+0000 7f01a489a640 10 monclient(hunting): _renew_subs 2024-03-29T10:47:05.748 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:05.765+0000 7f01a489a640 10 monclient(hunting): authenticate will time out at 1148.000786s 2024-03-29T10:47:05.748 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:05.766+0000 7f019e59c640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f01a00b0500 0x7f01a0072050 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:50203/0 (socket says 172.21.15.88:50203) 2024-03-29T10:47:05.748 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:05.766+0000 7f019e59c640 1 -- 172.21.15.88:0/1106075739 learned_addr learned my addr 172.21.15.88:0/1106075739 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-29T10:47:05.749 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:05.767+0000 7f019d59a640 1 -- 172.21.15.88:0/1106075739 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1500455266 0 0) 0x7f01a00ae7e0 con 0x7f01a00b0500 2024-03-29T10:47:05.749 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:05.767+0000 7f019d59a640 10 monclient(hunting): _negotiate 2024-03-29T10:47:05.749 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:05.767+0000 7f019d59a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:05.749 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:05.767+0000 7f019d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:05.749 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:05.767+0000 7f019d59a640 10 monclient(hunting): my global_id is 4455 2024-03-29T10:47:05.749 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:05.767+0000 7f019d59a640 1 -- 172.21.15.88:0/1106075739 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0188003520 con 0x7f01a00b0500 2024-03-29T10:47:05.749 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:05.767+0000 7f019d59a640 1 -- 172.21.15.88:0/1106075739 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1362957273 0 0) 0x7f0188003520 con 0x7f01a00b0500 2024-03-29T10:47:05.749 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:05.767+0000 7f019d59a640 1 -- 172.21.15.88:0/1106075739 >> v1:172.21.15.88:6790/0 conn(0x7f01a00b08e0 legacy=0x7f01a013d3e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:47:05.751 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:05.769+0000 7f019d59a640 1 -- 172.21.15.88:0/1106075739 >> v1:172.21.15.196:6789/0 conn(0x7f01a00aec70 legacy=0x7f01a013ff10 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-29T10:47:05.751 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:05.769+0000 7f019d59a640 10 monclient: _finish_hunting 0 2024-03-29T10:47:05.751 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:05.769+0000 7f019d59a640 1 monclient: found mon.noname-a 2024-03-29T10:47:05.751 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:05.769+0000 7f019d59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:47:05.751 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:05.769+0000 7f019d59a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.88:6789/0 2024-03-29T10:47:05.751 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:05.769+0000 7f019d59a640 1 -- 172.21.15.88:0/1106075739 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01a0140620 con 0x7f01a00b0500 2024-03-29T10:47:05.751 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:05.769+0000 7f019d59a640 10 monclient: _finish_auth 0 2024-03-29T10:47:05.751 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:05.769+0000 7f019d59a640 10 monclient: _check_auth_tickets 2024-03-29T10:47:05.751 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:05.769+0000 7f019d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:47:05.751 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:05.769+0000 7f019d59a640 1 -- 172.21.15.88:0/1106075739 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f0190001690 con 0x7f01a00b0500 2024-03-29T10:47:05.752 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:05.769+0000 7f019d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:05.752 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:05.769+0000 7f019d59a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:47:05.752 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:05.769+0000 7f019d59a640 10 monclient: dump: 2024-03-29T10:47:05.752 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:47:05.752 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:47:05.752 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:47:05.752 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:47:05.752 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:05.752 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:47:05.752 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:47:05.752 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:47:05.752 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:47:05.752 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:47:05.752 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:05.769+0000 7f01a489a640 5 monclient: authenticate success, global_id 4455 2024-03-29T10:47:05.752 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:05.769+0000 7f01a489a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:47:05.754 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:05.770+0000 7f019d59a640 1 -- 172.21.15.88:0/1106075739 <== mon.0 v1:172.21.15.88:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0190001860 con 0x7f01a00b0500 2024-03-29T10:47:05.754 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:05.770+0000 7f019d59a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:47:05.754 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:05.770+0000 7f019d59a640 1 -- 172.21.15.88:0/1106075739 <== mon.0 v1:172.21.15.88:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f0190001cb0 con 0x7f01a00b0500 2024-03-29T10:47:05.754 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:05.770+0000 7f019d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:05.754 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:05.770+0000 7f019d59a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:47:05.754 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:05.770+0000 7f019d59a640 10 monclient: dump: 2024-03-29T10:47:05.754 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:47:05.755 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:47:05.755 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:47:05.755 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:47:05.755 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:05.755 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:47:05.755 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:47:05.755 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:47:05.755 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:47:05.755 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:47:05.755 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:05.770+0000 7f01a489a640 10 monclient: get_monmap_and_config success 2024-03-29T10:47:05.755 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:05.770+0000 7f01a489a640 4 set_mon_vals no callback set 2024-03-29T10:47:05.755 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:05.771+0000 7f01a489a640 10 monclient: shutdown 2024-03-29T10:47:05.755 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:05.771+0000 7f01a489a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:47:05.755 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:05.771+0000 7f01a489a640 1 -- 172.21.15.88:0/1106075739 >> v1:172.21.15.88:6789/0 conn(0x7f01a00b0500 legacy=0x7f01a0072050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:05.756 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:05.771+0000 7f01a489a640 1 -- 172.21.15.88:0/1106075739 shutdown_connections 2024-03-29T10:47:05.756 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:05.771+0000 7f01a489a640 1 -- 172.21.15.88:0/1106075739 >> 172.21.15.88:0/1106075739 conn(0x7f01a006b660 msgr2=0x7f01a006ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:47:05.756 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:05.773+0000 7f01a489a640 1 -- 172.21.15.88:0/1106075739 shutdown_connections 2024-03-29T10:47:05.757 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:05.773+0000 7f01a489a640 1 -- 172.21.15.88:0/1106075739 wait complete. 2024-03-29T10:47:05.757 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:05.775+0000 7f01a489a640 10 monclient: build_initial_monmap 2024-03-29T10:47:05.757 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:05.775+0000 7f01a489a640 10 monclient: monmap: 2024-03-29T10:47:05.757 INFO:teuthology.misc.health.smithi088.stderr:epoch 0 2024-03-29T10:47:05.757 INFO:teuthology.misc.health.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:47:05.757 INFO:teuthology.misc.health.smithi088.stderr:last_changed 0.000000 2024-03-29T10:47:05.757 INFO:teuthology.misc.health.smithi088.stderr:created 0.000000 2024-03-29T10:47:05.757 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 0 (unknown) 2024-03-29T10:47:05.757 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:47:05.757 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-03-29T10:47:05.758 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-29T10:47:05.758 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-c 2024-03-29T10:47:05.758 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:47:05.758 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:05.775+0000 7f01a489a640 1 librados: starting msgr at 2024-03-29T10:47:05.758 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:05.775+0000 7f01a489a640 1 librados: starting objecter 2024-03-29T10:47:05.758 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:05.775+0000 7f01a489a640 1 Processor -- start 2024-03-29T10:47:05.759 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:05.775+0000 7f01a489a640 1 -- start start 2024-03-29T10:47:05.759 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:05.775+0000 7f01a489a640 1 librados: setting wanted keys 2024-03-29T10:47:05.759 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:05.775+0000 7f01a489a640 1 librados: calling monclient init 2024-03-29T10:47:05.759 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:05.775+0000 7f01a489a640 10 monclient: init 2024-03-29T10:47:05.759 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:05.776+0000 7f01a489a640 10 monclient: _reopen_session rank -1 2024-03-29T10:47:05.759 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:05.776+0000 7f01a489a640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:47:05.759 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:05.776+0000 7f01a489a640 10 monclient(hunting): picked mon.noname-c con 0x7f01a00aec70 addr v1:172.21.15.88:6790/0 2024-03-29T10:47:05.760 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:05.778+0000 7f01a489a640 10 monclient(hunting): picked mon.noname-b con 0x7f01a01e4180 addr v1:172.21.15.196:6789/0 2024-03-29T10:47:05.761 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:05.778+0000 7f01a489a640 10 monclient(hunting): picked mon.noname-a con 0x7f01a006eec0 addr v1:172.21.15.88:6789/0 2024-03-29T10:47:05.761 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:05.778+0000 7f01a489a640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01a01d4b70 con 0x7f01a006eec0 2024-03-29T10:47:05.761 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:05.778+0000 7f01a489a640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01a01e84a0 con 0x7f01a01e4180 2024-03-29T10:47:05.761 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:05.778+0000 7f01a489a640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01a006ff70 con 0x7f01a00aec70 2024-03-29T10:47:05.761 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:05.778+0000 7f01a489a640 10 monclient(hunting): _renew_subs 2024-03-29T10:47:05.761 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:05.778+0000 7f01a489a640 10 monclient(hunting): authenticate will time out at 1148.012966s 2024-03-29T10:47:05.761 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:05.780+0000 7f019dd9b640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f01a01e4180 0x7f01a01e7d90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.88:55987/0 (socket says 172.21.15.88:55987) 2024-03-29T10:47:05.762 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:05.780+0000 7f019dd9b640 1 -- 172.21.15.88:0/509709131 learned_addr learned my addr 172.21.15.88:0/509709131 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-29T10:47:05.763 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:05.781+0000 7f018effd640 10 client.?.objecter ms_handle_connect 0x7f01a01e4180 2024-03-29T10:47:05.763 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:05.781+0000 7f018effd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:47:05.763 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:05.781+0000 7f018effd640 10 client.?.objecter ms_handle_connect 0x7f01a00aec70 2024-03-29T10:47:05.763 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:05.781+0000 7f018effd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:47:05.764 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:05.781+0000 7f018effd640 1 -- 172.21.15.88:0/509709131 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 134357495 0 0) 0x7f01a006ff70 con 0x7f01a00aec70 2024-03-29T10:47:05.764 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:05.781+0000 7f018effd640 10 monclient(hunting): _negotiate 2024-03-29T10:47:05.764 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:05.781+0000 7f018effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:05.764 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:05.781+0000 7f018effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:05.764 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:05.781+0000 7f018effd640 10 monclient(hunting): my global_id is 4487 2024-03-29T10:47:05.764 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:05.781+0000 7f018effd640 1 -- 172.21.15.88:0/509709131 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0180004eb0 con 0x7f01a00aec70 2024-03-29T10:47:05.768 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:05.783+0000 7f018effd640 1 -- 172.21.15.88:0/509709131 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 262819486 0 0) 0x7f01a01e84a0 con 0x7f01a01e4180 2024-03-29T10:47:05.768 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:05.783+0000 7f018effd640 10 monclient(hunting): _negotiate 2024-03-29T10:47:05.768 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:05.783+0000 7f018effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:05.768 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:05.783+0000 7f018effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:05.768 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:05.783+0000 7f018effd640 10 monclient(hunting): my global_id is 4450 2024-03-29T10:47:05.768 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:05.783+0000 7f018effd640 1 -- 172.21.15.88:0/509709131 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f01a006ff70 con 0x7f01a01e4180 2024-03-29T10:47:05.768 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:05.783+0000 7f018effd640 10 client.?.objecter ms_handle_connect 0x7f01a006eec0 2024-03-29T10:47:05.769 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:05.783+0000 7f018effd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:47:05.769 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:05.783+0000 7f018effd640 1 -- 172.21.15.88:0/509709131 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1118815841 0 0) 0x7f01a01d4b70 con 0x7f01a006eec0 2024-03-29T10:47:05.769 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:05.783+0000 7f018effd640 10 monclient(hunting): _negotiate 2024-03-29T10:47:05.769 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:05.783+0000 7f018effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:05.769 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:05.783+0000 7f018effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:05.769 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:05.783+0000 7f018effd640 10 monclient(hunting): my global_id is 4458 2024-03-29T10:47:05.769 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:05.783+0000 7f018effd640 1 -- 172.21.15.88:0/509709131 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f01a01e84a0 con 0x7f01a006eec0 2024-03-29T10:47:05.769 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:05.784+0000 7f018effd640 1 -- 172.21.15.88:0/509709131 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3535063525 0 0) 0x7f01a01e84a0 con 0x7f01a006eec0 2024-03-29T10:47:05.769 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:05.784+0000 7f018effd640 1 -- 172.21.15.88:0/509709131 --> v1:172.21.15.88:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f01a01d4b70 con 0x7f01a006eec0 2024-03-29T10:47:05.769 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:05.784+0000 7f018effd640 1 -- 172.21.15.88:0/509709131 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f0198001a70 con 0x7f01a006eec0 2024-03-29T10:47:05.769 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:05.784+0000 7f018effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:05.769 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:05.784+0000 7f018effd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:47:05.769 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:05.784+0000 7f018effd640 10 monclient(hunting): dump: 2024-03-29T10:47:05.769 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:47:05.770 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:47:05.770 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:47:05.770 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:47:05.770 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:05.770 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:47:05.770 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:47:05.770 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:47:05.770 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:47:05.770 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:47:05.770 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:05.784+0000 7f018effd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:47:05.770 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:05.784+0000 7f018effd640 1 -- 172.21.15.88:0/509709131 <== mon.0 v1:172.21.15.88:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 349790512 0 0) 0x7f01a01d4b70 con 0x7f01a006eec0 2024-03-29T10:47:05.770 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:05.784+0000 7f018effd640 1 -- 172.21.15.88:0/509709131 >> v1:172.21.15.88:6790/0 conn(0x7f01a00aec70 legacy=0x7f01a01e7680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:05.770 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:05.784+0000 7f018effd640 1 -- 172.21.15.88:0/509709131 >> v1:172.21.15.196:6789/0 conn(0x7f01a01e4180 legacy=0x7f01a01e7d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:05.770 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:05.785+0000 7f018effd640 10 monclient: _finish_hunting 0 2024-03-29T10:47:05.770 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:05.785+0000 7f018effd640 1 monclient: found mon.a 2024-03-29T10:47:05.771 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:05.785+0000 7f018effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:47:05.771 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:05.785+0000 7f018effd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-03-29T10:47:05.771 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:05.785+0000 7f018effd640 1 -- 172.21.15.88:0/509709131 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01a0070250 con 0x7f01a006eec0 2024-03-29T10:47:05.771 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:05.785+0000 7f018effd640 10 monclient: _finish_auth 0 2024-03-29T10:47:05.771 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:05.785+0000 7f018effd640 10 monclient: _check_auth_tickets 2024-03-29T10:47:05.771 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:05.785+0000 7f018effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:47:05.771 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:05.785+0000 7f01a489a640 5 monclient: authenticate success, global_id 4458 2024-03-29T10:47:05.771 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:05.785+0000 7f01a489a640 10 monclient: _renew_subs 2024-03-29T10:47:05.771 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:05.785+0000 7f01a489a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-03-29T10:47:05.771 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:05.785+0000 7f01a489a640 1 -- 172.21.15.88:0/509709131 --> v1:172.21.15.88:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01a01e46a0 con 0x7f01a006eec0 2024-03-29T10:47:05.771 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:05.785+0000 7f01a489a640 10 client.4458.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:47:05.771 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:05.785+0000 7f01a489a640 10 monclient: _renew_subs 2024-03-29T10:47:05.771 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:05.785+0000 7f01a489a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-03-29T10:47:05.771 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:05.785+0000 7f01a489a640 1 -- 172.21.15.88:0/509709131 --> v1:172.21.15.88:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f01a00709e0 con 0x7f01a006eec0 2024-03-29T10:47:05.772 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:05.785+0000 7f01a489a640 1 librados: init done 2024-03-29T10:47:05.772 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:05.786+0000 7f01a489a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:05.772 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:05.786+0000 7f01a489a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:05.772 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:05.787+0000 7f01a489a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-03-29T10:47:05.772 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:05.787+0000 7f01a489a640 1 -- 172.21.15.88:0/509709131 --> v1:172.21.15.88:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01a00af580 con 0x7f01a006eec0 2024-03-29T10:47:05.773 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:05.790+0000 7f018effd640 1 -- 172.21.15.88:0/509709131 <== mon.0 v1:172.21.15.88:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0198001c50 con 0x7f01a006eec0 2024-03-29T10:47:05.773 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:05.790+0000 7f018effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:47:05.773 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:05.790+0000 7f018effd640 1 -- 172.21.15.88:0/509709131 <== mon.0 v1:172.21.15.88:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f0198002180 con 0x7f01a006eec0 2024-03-29T10:47:05.773 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:05.790+0000 7f018effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:05.773 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:05.790+0000 7f018effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:47:05.774 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:05.790+0000 7f018effd640 10 monclient: dump: 2024-03-29T10:47:05.774 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:47:05.774 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:47:05.774 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:47:05.774 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:47:05.774 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:05.774 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:47:05.774 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:47:05.774 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:47:05.774 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:47:05.774 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:47:05.774 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:05.791+0000 7f019cd99640 4 set_mon_vals no callback set 2024-03-29T10:47:05.774 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:05.791+0000 7f018effd640 1 -- 172.21.15.88:0/509709131 <== mon.0 v1:172.21.15.88:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4235629220 0 0) 0x7f0198019ae0 con 0x7f01a006eec0 2024-03-29T10:47:05.775 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:05.792+0000 7f019e59c640 10 monclient: get_auth_request con 0x7f018006de80 auth_method 0 2024-03-29T10:47:05.775 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:05.793+0000 7f018effd640 1 -- 172.21.15.88:0/509709131 <== mon.0 v1:172.21.15.88:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4257673582 0 0) 0x7f0198083a10 con 0x7f01a006eec0 2024-03-29T10:47:05.775 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:05.793+0000 7f018effd640 10 client.4458.objecter ms_dispatch 0x7f01a0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:47:05.776 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:05.793+0000 7f018effd640 3 client.4458.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:47:05.776 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:05.793+0000 7f018effd640 3 client.4458.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:47:05.776 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:05.793+0000 7f018effd640 20 client.4458.objecter dump_active .. 0 homeless 2024-03-29T10:47:05.776 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:05.793+0000 7f018effd640 10 client.4458.objecter ms_handle_connect 0x7f018006de80 2024-03-29T10:47:05.783 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:05.802+0000 7f018effd640 1 -- 172.21.15.88:0/509709131 <== mon.0 v1:172.21.15.88:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f019804f2f0 con 0x7f01a006eec0 2024-03-29T10:47:05.783 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:05.802+0000 7f018effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:05.783 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:05.802+0000 7f018effd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:47:06.101 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:06.119+0000 7f01a489a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:47:06.101 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:06.119+0000 7f01a489a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:47:06.101 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:06.119+0000 7f01a489a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-03-29T10:47:06.101 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:06.119+0000 7f01a489a640 1 -- 172.21.15.88:0/509709131 --> v1:172.21.15.88:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f01a0070d30 con 0x7f01a006eec0 2024-03-29T10:47:06.101 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:06.119+0000 7f018effd640 1 -- 172.21.15.88:0/509709131 <== mon.0 v1:172.21.15.88:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 344506374) 0x7f0198051220 con 0x7f01a006eec0 2024-03-29T10:47:06.101 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:06.119+0000 7f018effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:47:06.101 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:06.119+0000 7f018effd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:47:06.102 INFO:teuthology.misc.health.smithi088.stdout: 2024-03-29T10:47:06.102 INFO:teuthology.misc.health.smithi088.stdout:{"epoch":10,"fsid":"869f5188-f92b-42dd-a46d-8c646e0388a9","created":"2024-03-29T10:40:40.285087+0000","modified":"2024-03-29T10:40:42.536230+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:40:42.536230+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a26c0c1c-8de0-438c-a3f6-3df10ea61574","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5de24915-8dec-49b6-8b16-f3170197c591","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ac054af3-d589-4565-9644-e6af75975c53","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3ec4f045-c714-4a22-bd1a-e07330b45ab3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"cddb9867-8e30-4951-9b13-3d1e78b9c45a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dc6d9911-785d-432f-8b53-56a1ccab302f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"77c88437-b587-496e-8b4d-1824ea8b5305","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ce963d96-5a8e-44c3-99de-c0dbebe234ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:47:06.109 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:06.124+0000 7f01a489a640 10 librados: watch_flush enter 2024-03-29T10:47:06.109 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:06.124+0000 7f01a489a640 10 librados: watch_flush exit 2024-03-29T10:47:06.109 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:06.124+0000 7f01a489a640 20 client.4458.objecter shutdown clearing up homeless session... 2024-03-29T10:47:06.109 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:06.124+0000 7f01a489a640 10 client.4458.objecter successfully canceled tick 2024-03-29T10:47:06.109 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:06.124+0000 7f01a489a640 1 -- 172.21.15.88:0/509709131 >> v1:172.21.15.88:6816/1752861349 conn(0x7f018006de80 legacy=0x7f0180070340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:06.109 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:06.124+0000 7f01a489a640 10 monclient: shutdown 2024-03-29T10:47:06.109 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:06.124+0000 7f01a489a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:47:06.109 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:06.124+0000 7f01a489a640 1 -- 172.21.15.88:0/509709131 >> v1:172.21.15.88:6789/0 conn(0x7f01a006eec0 legacy=0x7f01a01ef9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:06.109 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:06.126+0000 7f01a489a640 1 -- 172.21.15.88:0/509709131 shutdown_connections 2024-03-29T10:47:06.109 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:06.127+0000 7f01a489a640 1 -- 172.21.15.88:0/509709131 >> 172.21.15.88:0/509709131 conn(0x7f01a006b660 msgr2=0x7f01a0071e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:47:06.110 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:06.127+0000 7f01a489a640 1 -- 172.21.15.88:0/509709131 shutdown_connections 2024-03-29T10:47:06.110 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:06.127+0000 7f01a489a640 1 -- 172.21.15.88:0/509709131 wait complete. 2024-03-29T10:47:06.110 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:06.127+0000 7f01a489a640 1 librados: shutdown 2024-03-29T10:47:06.125 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:47:12.931 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:47:13.079 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:13.095+0000 7fd8f7b0e640 10 monclient: get_monmap_and_config 2024-03-29T10:47:13.079 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:13.095+0000 7fd8f7b0e640 10 monclient: build_initial_monmap 2024-03-29T10:47:13.079 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:13.095+0000 7fd8f7b0e640 10 monclient: monmap: 2024-03-29T10:47:13.079 INFO:teuthology.misc.health.smithi088.stderr:epoch 0 2024-03-29T10:47:13.079 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:47:13.079 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:47:13.096923+0000 2024-03-29T10:47:13.079 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:47:13.096923+0000 2024-03-29T10:47:13.079 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 0 (unknown) 2024-03-29T10:47:13.079 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:47:13.079 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-03-29T10:47:13.080 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-29T10:47:13.080 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-b 2024-03-29T10:47:13.080 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:47:13.081 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:13.099+0000 7fd8f7b0e640 1 Processor -- start 2024-03-29T10:47:13.082 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:13.100+0000 7fd8f7b0e640 1 -- start start 2024-03-29T10:47:13.082 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:13.100+0000 7fd8f7b0e640 10 monclient: init 2024-03-29T10:47:13.083 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:13.101+0000 7fd8f7b0e640 10 monclient: _reopen_session rank -1 2024-03-29T10:47:13.088 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:13.101+0000 7fd8f7b0e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:47:13.089 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:13.101+0000 7fd8f7b0e640 10 monclient(hunting): picked mon.noname-c con 0x7fd8f00a9ac0 addr v1:172.21.15.196:6789/0 2024-03-29T10:47:13.089 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:13.101+0000 7fd8f7b0e640 10 monclient(hunting): picked mon.noname-b con 0x7fd8f013ee90 addr v1:172.21.15.88:6790/0 2024-03-29T10:47:13.089 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:13.101+0000 7fd8f7b0e640 10 monclient(hunting): picked mon.noname-a con 0x7fd8f00af900 addr v1:172.21.15.88:6789/0 2024-03-29T10:47:13.089 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:13.101+0000 7fd8f7b0e640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8f00afd10 con 0x7fd8f00af900 2024-03-29T10:47:13.089 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:13.101+0000 7fd8f7b0e640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8f00af6f0 con 0x7fd8f00a9ac0 2024-03-29T10:47:13.089 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:13.101+0000 7fd8f7b0e640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8f00aac80 con 0x7fd8f013ee90 2024-03-29T10:47:13.089 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:13.101+0000 7fd8f7b0e640 10 monclient(hunting): _renew_subs 2024-03-29T10:47:13.089 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:13.101+0000 7fd8f7b0e640 10 monclient(hunting): authenticate will time out at 1155.337533s 2024-03-29T10:47:13.089 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:13.102+0000 7fd8f50a9640 1 --1- >> v1:172.21.15.88:6790/0 conn(0x7fd8f013ee90 0x7fd8f013b5f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6790/0 says I am v1:172.21.15.88:32960/0 (socket says 172.21.15.88:32960) 2024-03-29T10:47:13.089 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:13.102+0000 7fd8f50a9640 1 -- 172.21.15.88:0/2755169393 learned_addr learned my addr 172.21.15.88:0/2755169393 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-29T10:47:13.089 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:13.105+0000 7fd8f48a8640 1 -- 172.21.15.88:0/2755169393 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3876054284 0 0) 0x7fd8f00af6f0 con 0x7fd8f00a9ac0 2024-03-29T10:47:13.089 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:13.105+0000 7fd8f48a8640 10 monclient(hunting): _negotiate 2024-03-29T10:47:13.090 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:13.105+0000 7fd8f48a8640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:13.090 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:13.105+0000 7fd8f48a8640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:13.090 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:13.105+0000 7fd8f48a8640 10 monclient(hunting): my global_id is 4453 2024-03-29T10:47:13.090 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:13.105+0000 7fd8f48a8640 1 -- 172.21.15.88:0/2755169393 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd8dc003520 con 0x7fd8f00a9ac0 2024-03-29T10:47:13.090 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:13.106+0000 7fd8f48a8640 1 -- 172.21.15.88:0/2755169393 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3136480509 0 0) 0x7fd8f00aac80 con 0x7fd8f013ee90 2024-03-29T10:47:13.090 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:13.106+0000 7fd8f48a8640 10 monclient(hunting): _negotiate 2024-03-29T10:47:13.090 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:13.106+0000 7fd8f48a8640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:13.091 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:13.106+0000 7fd8f48a8640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:13.091 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:13.106+0000 7fd8f48a8640 10 monclient(hunting): my global_id is 4490 2024-03-29T10:47:13.091 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:13.106+0000 7fd8f48a8640 1 -- 172.21.15.88:0/2755169393 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd8f00af6f0 con 0x7fd8f013ee90 2024-03-29T10:47:13.091 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:13.106+0000 7fd8f48a8640 1 -- 172.21.15.88:0/2755169393 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2396858082 0 0) 0x7fd8f00af6f0 con 0x7fd8f013ee90 2024-03-29T10:47:13.091 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:13.106+0000 7fd8f48a8640 1 -- 172.21.15.88:0/2755169393 >> v1:172.21.15.196:6789/0 conn(0x7fd8f00a9ac0 legacy=0x7fd8f00adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:13.091 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:13.107+0000 7fd8f48a8640 1 -- 172.21.15.88:0/2755169393 >> v1:172.21.15.88:6789/0 conn(0x7fd8f00af900 legacy=0x7fd8f0149b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:13.091 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:13.107+0000 7fd8f48a8640 10 monclient: _finish_hunting 0 2024-03-29T10:47:13.092 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:13.107+0000 7fd8f48a8640 1 monclient: found mon.noname-b 2024-03-29T10:47:13.092 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:13.107+0000 7fd8f48a8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:47:13.092 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:13.107+0000 7fd8f48a8640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.88:6790/0 2024-03-29T10:47:13.092 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:13.107+0000 7fd8f48a8640 1 -- 172.21.15.88:0/2755169393 --> v1:172.21.15.88:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8f00aaf60 con 0x7fd8f013ee90 2024-03-29T10:47:13.092 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:13.107+0000 7fd8f48a8640 10 monclient: _finish_auth 0 2024-03-29T10:47:13.092 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:13.107+0000 7fd8f48a8640 10 monclient: _check_auth_tickets 2024-03-29T10:47:13.092 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:13.107+0000 7fd8f48a8640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:47:13.092 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:13.107+0000 7fd8f48a8640 1 -- 172.21.15.88:0/2755169393 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7fd8e8004fd0 con 0x7fd8f013ee90 2024-03-29T10:47:13.092 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:13.107+0000 7fd8f48a8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:13.092 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:13.107+0000 7fd8f48a8640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:47:13.092 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:13.107+0000 7fd8f48a8640 10 monclient: dump: 2024-03-29T10:47:13.092 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:47:13.092 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:47:13.092 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:47:13.092 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:47:13.093 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:13.093 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:47:13.093 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:47:13.093 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:47:13.093 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:47:13.093 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:47:13.093 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:13.107+0000 7fd8f7b0e640 5 monclient: authenticate success, global_id 4490 2024-03-29T10:47:13.093 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:13.107+0000 7fd8f7b0e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:47:13.093 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:13.107+0000 7fd8f48a8640 1 -- 172.21.15.88:0/2755169393 <== mon.2 v1:172.21.15.88:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd8e80051a0 con 0x7fd8f013ee90 2024-03-29T10:47:13.093 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:13.107+0000 7fd8f48a8640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:47:13.093 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:13.107+0000 7fd8f7b0e640 10 monclient: get_monmap_and_config success 2024-03-29T10:47:13.093 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:13.107+0000 7fd8f7b0e640 4 set_mon_vals no callback set 2024-03-29T10:47:13.093 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:13.108+0000 7fd8f7b0e640 10 monclient: shutdown 2024-03-29T10:47:13.094 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:13.108+0000 7fd8f7b0e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:47:13.094 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:13.108+0000 7fd8f7b0e640 1 -- 172.21.15.88:0/2755169393 >> v1:172.21.15.88:6790/0 conn(0x7fd8f013ee90 legacy=0x7fd8f013b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:13.094 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:13.108+0000 7fd8f7b0e640 1 -- 172.21.15.88:0/2755169393 shutdown_connections 2024-03-29T10:47:13.094 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:13.108+0000 7fd8f7b0e640 1 -- 172.21.15.88:0/2755169393 >> 172.21.15.88:0/2755169393 conn(0x7fd8f00b35e0 msgr2=0x7fd8f00b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:47:13.094 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:13.109+0000 7fd8f7b0e640 1 -- 172.21.15.88:0/2755169393 shutdown_connections 2024-03-29T10:47:13.096 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:13.109+0000 7fd8f7b0e640 1 -- 172.21.15.88:0/2755169393 wait complete. 2024-03-29T10:47:13.096 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:13.110+0000 7fd8f7b0e640 10 monclient: build_initial_monmap 2024-03-29T10:47:13.096 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:13.110+0000 7fd8f7b0e640 10 monclient: monmap: 2024-03-29T10:47:13.096 INFO:teuthology.misc.health.smithi088.stderr:epoch 0 2024-03-29T10:47:13.097 INFO:teuthology.misc.health.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:47:13.097 INFO:teuthology.misc.health.smithi088.stderr:last_changed 0.000000 2024-03-29T10:47:13.097 INFO:teuthology.misc.health.smithi088.stderr:created 0.000000 2024-03-29T10:47:13.097 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 0 (unknown) 2024-03-29T10:47:13.097 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:47:13.097 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-03-29T10:47:13.097 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-29T10:47:13.097 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-c 2024-03-29T10:47:13.097 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:47:13.097 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:13.110+0000 7fd8f7b0e640 1 librados: starting msgr at 2024-03-29T10:47:13.097 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:13.110+0000 7fd8f7b0e640 1 librados: starting objecter 2024-03-29T10:47:13.097 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:13.110+0000 7fd8f7b0e640 1 Processor -- start 2024-03-29T10:47:13.098 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:13.110+0000 7fd8f7b0e640 1 -- start start 2024-03-29T10:47:13.098 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:13.110+0000 7fd8f7b0e640 1 librados: setting wanted keys 2024-03-29T10:47:13.098 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:13.110+0000 7fd8f7b0e640 1 librados: calling monclient init 2024-03-29T10:47:13.098 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:13.110+0000 7fd8f7b0e640 10 monclient: init 2024-03-29T10:47:13.098 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:13.111+0000 7fd8f7b0e640 10 monclient: _reopen_session rank -1 2024-03-29T10:47:13.098 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:13.111+0000 7fd8f7b0e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:47:13.098 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:13.111+0000 7fd8f7b0e640 10 monclient(hunting): picked mon.noname-a con 0x7fd8f00a9ac0 addr v1:172.21.15.88:6789/0 2024-03-29T10:47:13.098 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:13.111+0000 7fd8f7b0e640 10 monclient(hunting): picked mon.noname-c con 0x7fd8f00af900 addr v1:172.21.15.88:6790/0 2024-03-29T10:47:13.098 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:13.111+0000 7fd8f7b0e640 10 monclient(hunting): picked mon.noname-b con 0x7fd8f013ee90 addr v1:172.21.15.196:6789/0 2024-03-29T10:47:13.098 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:13.111+0000 7fd8f7b0e640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8f01e8160 con 0x7fd8f00a9ac0 2024-03-29T10:47:13.098 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:13.111+0000 7fd8f7b0e640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8f01e82f0 con 0x7fd8f013ee90 2024-03-29T10:47:13.098 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:13.111+0000 7fd8f7b0e640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8f01c4ae0 con 0x7fd8f00af900 2024-03-29T10:47:13.099 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:13.111+0000 7fd8f7b0e640 10 monclient(hunting): _renew_subs 2024-03-29T10:47:13.099 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:13.111+0000 7fd8f7b0e640 10 monclient(hunting): authenticate will time out at 1155.347640s 2024-03-29T10:47:13.099 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:13.114+0000 7fd8f50a9640 1 --1- >> v1:172.21.15.88:6790/0 conn(0x7fd8f00af900 0x7fd8f01c43d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6790/0 says I am v1:172.21.15.88:34760/0 (socket says 172.21.15.88:34760) 2024-03-29T10:47:13.099 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:13.114+0000 7fd8f50a9640 1 -- 172.21.15.88:0/1214572285 learned_addr learned my addr 172.21.15.88:0/1214572285 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-29T10:47:13.101 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:13.114+0000 7fd8e67fc640 10 client.?.objecter ms_handle_connect 0x7fd8f00af900 2024-03-29T10:47:13.101 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:13.114+0000 7fd8e67fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:47:13.101 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:13.114+0000 7fd8e67fc640 1 -- 172.21.15.88:0/1214572285 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 700867829 0 0) 0x7fd8f01c4ae0 con 0x7fd8f00af900 2024-03-29T10:47:13.102 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:13.114+0000 7fd8e67fc640 10 monclient(hunting): _negotiate 2024-03-29T10:47:13.102 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:13.114+0000 7fd8e67fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:13.102 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:13.114+0000 7fd8e67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:13.102 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:13.114+0000 7fd8e67fc640 10 monclient(hunting): my global_id is 4493 2024-03-29T10:47:13.102 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:13.114+0000 7fd8e67fc640 1 -- 172.21.15.88:0/1214572285 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd8d4004eb0 con 0x7fd8f00af900 2024-03-29T10:47:13.102 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:13.115+0000 7fd8e67fc640 1 -- 172.21.15.88:0/1214572285 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1755816027 0 0) 0x7fd8d4004eb0 con 0x7fd8f00af900 2024-03-29T10:47:13.102 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:13.115+0000 7fd8e67fc640 1 -- 172.21.15.88:0/1214572285 --> v1:172.21.15.88:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd8f01c4ae0 con 0x7fd8f00af900 2024-03-29T10:47:13.102 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:13.115+0000 7fd8e67fc640 1 -- 172.21.15.88:0/1214572285 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7fd8e8001c30 con 0x7fd8f00af900 2024-03-29T10:47:13.102 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:13.115+0000 7fd8e67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:13.102 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:13.115+0000 7fd8e67fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:47:13.102 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:13.115+0000 7fd8e67fc640 10 monclient(hunting): dump: 2024-03-29T10:47:13.102 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:47:13.102 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:47:13.103 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:47:13.103 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:47:13.103 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:13.103 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:47:13.103 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:47:13.103 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:47:13.103 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:47:13.103 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:47:13.103 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:13.115+0000 7fd8e67fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:47:13.103 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:13.116+0000 7fd8e67fc640 1 -- 172.21.15.88:0/1214572285 <== mon.2 v1:172.21.15.88:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 466345231 0 0) 0x7fd8f01c4ae0 con 0x7fd8f00af900 2024-03-29T10:47:13.103 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:13.116+0000 7fd8e67fc640 1 -- 172.21.15.88:0/1214572285 >> v1:172.21.15.196:6789/0 conn(0x7fd8f013ee90 legacy=0x7fd8f01ee9d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:47:13.103 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:13.117+0000 7fd8e67fc640 1 -- 172.21.15.88:0/1214572285 >> v1:172.21.15.88:6789/0 conn(0x7fd8f00a9ac0 legacy=0x7fd8f01e7700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:13.103 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:13.117+0000 7fd8e67fc640 10 monclient: _finish_hunting 0 2024-03-29T10:47:13.103 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:13.117+0000 7fd8e67fc640 1 monclient: found mon.c 2024-03-29T10:47:13.104 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:13.117+0000 7fd8e67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:47:13.104 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:13.117+0000 7fd8e67fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-03-29T10:47:13.104 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:13.117+0000 7fd8e67fc640 1 -- 172.21.15.88:0/1214572285 --> v1:172.21.15.88:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8f01c4c70 con 0x7fd8f00af900 2024-03-29T10:47:13.104 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:13.117+0000 7fd8e67fc640 10 monclient: _finish_auth 0 2024-03-29T10:47:13.104 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:13.117+0000 7fd8e67fc640 10 monclient: _check_auth_tickets 2024-03-29T10:47:13.104 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:13.117+0000 7fd8e67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:47:13.104 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:13.118+0000 7fd8e67fc640 10 client.?.objecter ms_handle_connect 0x7fd8f00a9ac0 2024-03-29T10:47:13.104 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:13.118+0000 7fd8e67fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:47:13.104 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:13.118+0000 7fd8f7b0e640 5 monclient: authenticate success, global_id 4493 2024-03-29T10:47:13.104 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:13.118+0000 7fd8f7b0e640 10 monclient: _renew_subs 2024-03-29T10:47:13.104 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:13.118+0000 7fd8f7b0e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-03-29T10:47:13.104 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:13.118+0000 7fd8f7b0e640 1 -- 172.21.15.88:0/1214572285 --> v1:172.21.15.88:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8f0071780 con 0x7fd8f00af900 2024-03-29T10:47:13.104 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:13.118+0000 7fd8f7b0e640 10 client.4493.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:47:13.104 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:13.118+0000 7fd8f7b0e640 10 monclient: _renew_subs 2024-03-29T10:47:13.104 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:13.118+0000 7fd8f7b0e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-03-29T10:47:13.105 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:13.118+0000 7fd8f7b0e640 1 -- 172.21.15.88:0/1214572285 --> v1:172.21.15.88:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8f0072260 con 0x7fd8f00af900 2024-03-29T10:47:13.105 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:13.118+0000 7fd8f7b0e640 1 librados: init done 2024-03-29T10:47:13.106 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:13.119+0000 7fd8e67fc640 1 -- 172.21.15.88:0/1214572285 <== mon.2 v1:172.21.15.88:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd8e8002010 con 0x7fd8f00af900 2024-03-29T10:47:13.106 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:13.119+0000 7fd8e67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:47:13.106 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:13.119+0000 7fd8e67fc640 1 -- 172.21.15.88:0/1214572285 <== mon.2 v1:172.21.15.88:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7fd8e80015d0 con 0x7fd8f00af900 2024-03-29T10:47:13.106 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:13.119+0000 7fd8e7fff640 4 set_mon_vals no callback set 2024-03-29T10:47:13.106 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:13.119+0000 7fd8f7b0e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:13.106 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:13.121+0000 7fd8f7b0e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:13.107 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:13.121+0000 7fd8f7b0e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-03-29T10:47:13.107 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:13.121+0000 7fd8f7b0e640 1 -- 172.21.15.88:0/1214572285 --> v1:172.21.15.88:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8d8005080 con 0x7fd8f00af900 2024-03-29T10:47:13.107 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:13.122+0000 7fd8e67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:13.107 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:13.122+0000 7fd8e67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:47:13.107 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:13.122+0000 7fd8e67fc640 10 monclient: dump: 2024-03-29T10:47:13.107 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:47:13.107 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:47:13.107 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:47:13.107 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:47:13.107 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:13.107 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:47:13.107 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:47:13.107 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:47:13.107 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:47:13.108 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:47:13.108 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:13.122+0000 7fd8e67fc640 1 -- 172.21.15.88:0/1214572285 <== mon.2 v1:172.21.15.88:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4235629220 0 0) 0x7fd8e801b430 con 0x7fd8f00af900 2024-03-29T10:47:13.108 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:13.123+0000 7fd8e67fc640 1 -- 172.21.15.88:0/1214572285 <== mon.2 v1:172.21.15.88:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4257673582 0 0) 0x7fd8e80850e0 con 0x7fd8f00af900 2024-03-29T10:47:13.108 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:13.123+0000 7fd8e67fc640 10 client.4493.objecter ms_dispatch 0x7fd8f0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:47:13.108 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:13.123+0000 7fd8f58aa640 10 monclient: get_auth_request con 0x7fd8d406dbf0 auth_method 0 2024-03-29T10:47:13.108 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:13.123+0000 7fd8e67fc640 3 client.4493.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:47:13.108 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:13.123+0000 7fd8e67fc640 3 client.4493.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:47:13.108 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:13.124+0000 7fd8e67fc640 20 client.4493.objecter dump_active .. 0 homeless 2024-03-29T10:47:13.108 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:13.124+0000 7fd8e67fc640 10 client.4493.objecter ms_handle_connect 0x7fd8d406dbf0 2024-03-29T10:47:13.112 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:13.130+0000 7fd8e67fc640 1 -- 172.21.15.88:0/1214572285 <== mon.2 v1:172.21.15.88:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd8e80509c0 con 0x7fd8f00af900 2024-03-29T10:47:13.113 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:13.130+0000 7fd8e67fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:13.113 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:13.130+0000 7fd8e67fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:47:13.441 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:13.458+0000 7fd8f7b0e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:47:13.441 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:13.458+0000 7fd8f7b0e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:47:13.441 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:13.458+0000 7fd8f7b0e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-03-29T10:47:13.441 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:13.458+0000 7fd8f7b0e640 1 -- 172.21.15.88:0/1214572285 --> v1:172.21.15.88:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd8d8005330 con 0x7fd8f00af900 2024-03-29T10:47:13.441 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:13.459+0000 7fd8e67fc640 1 -- 172.21.15.88:0/1214572285 <== mon.2 v1:172.21.15.88:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 344506374) 0x7fd8e8001730 con 0x7fd8f00af900 2024-03-29T10:47:13.441 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:13.459+0000 7fd8e67fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:47:13.442 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:13.459+0000 7fd8e67fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:47:13.442 INFO:teuthology.misc.health.smithi088.stdout: 2024-03-29T10:47:13.442 INFO:teuthology.misc.health.smithi088.stdout:{"epoch":10,"fsid":"869f5188-f92b-42dd-a46d-8c646e0388a9","created":"2024-03-29T10:40:40.285087+0000","modified":"2024-03-29T10:40:42.536230+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:40:42.536230+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a26c0c1c-8de0-438c-a3f6-3df10ea61574","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5de24915-8dec-49b6-8b16-f3170197c591","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ac054af3-d589-4565-9644-e6af75975c53","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3ec4f045-c714-4a22-bd1a-e07330b45ab3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"cddb9867-8e30-4951-9b13-3d1e78b9c45a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dc6d9911-785d-432f-8b53-56a1ccab302f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"77c88437-b587-496e-8b4d-1824ea8b5305","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ce963d96-5a8e-44c3-99de-c0dbebe234ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:47:13.445 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:13.463+0000 7fd8f7b0e640 10 librados: watch_flush enter 2024-03-29T10:47:13.447 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:13.464+0000 7fd8f7b0e640 10 librados: watch_flush exit 2024-03-29T10:47:13.447 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:13.464+0000 7fd8f7b0e640 20 client.4493.objecter shutdown clearing up homeless session... 2024-03-29T10:47:13.447 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:13.464+0000 7fd8f7b0e640 10 client.4493.objecter successfully canceled tick 2024-03-29T10:47:13.447 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:13.465+0000 7fd8f7b0e640 1 -- 172.21.15.88:0/1214572285 >> v1:172.21.15.88:6816/1752861349 conn(0x7fd8d406dbf0 legacy=0x7fd8d40700b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:13.448 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:13.465+0000 7fd8f7b0e640 10 monclient: shutdown 2024-03-29T10:47:13.448 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:13.465+0000 7fd8f7b0e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:47:13.448 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:13.465+0000 7fd8f7b0e640 1 -- 172.21.15.88:0/1214572285 >> v1:172.21.15.88:6790/0 conn(0x7fd8f00af900 legacy=0x7fd8f01c43d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:13.448 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:13.465+0000 7fd8f7b0e640 1 -- 172.21.15.88:0/1214572285 shutdown_connections 2024-03-29T10:47:13.448 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:13.465+0000 7fd8f7b0e640 1 -- 172.21.15.88:0/1214572285 >> 172.21.15.88:0/1214572285 conn(0x7fd8f00b35e0 msgr2=0x7fd8f01ccf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:47:13.448 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:13.465+0000 7fd8f7b0e640 1 -- 172.21.15.88:0/1214572285 shutdown_connections 2024-03-29T10:47:13.448 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:13.465+0000 7fd8f7b0e640 1 -- 172.21.15.88:0/1214572285 wait complete. 2024-03-29T10:47:13.448 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:13.465+0000 7fd8f7b0e640 1 librados: shutdown 2024-03-29T10:47:13.465 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:47:20.273 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:47:20.423 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:20.441+0000 7f5729263640 10 monclient: get_monmap_and_config 2024-03-29T10:47:20.423 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:20.441+0000 7f5729263640 10 monclient: build_initial_monmap 2024-03-29T10:47:20.424 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:20.441+0000 7f5729263640 10 monclient: monmap: 2024-03-29T10:47:20.424 INFO:teuthology.misc.health.smithi088.stderr:epoch 0 2024-03-29T10:47:20.424 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:47:20.424 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:47:20.442085+0000 2024-03-29T10:47:20.424 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:47:20.442085+0000 2024-03-29T10:47:20.424 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 0 (unknown) 2024-03-29T10:47:20.424 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:47:20.424 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-03-29T10:47:20.424 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-29T10:47:20.424 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-b 2024-03-29T10:47:20.424 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:47:20.426 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:20.441+0000 7f5729263640 1 Processor -- start 2024-03-29T10:47:20.426 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:20.442+0000 7f5729263640 1 -- start start 2024-03-29T10:47:20.426 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:20.442+0000 7f5729263640 10 monclient: init 2024-03-29T10:47:20.426 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:20.443+0000 7f5729263640 10 monclient: _reopen_session rank -1 2024-03-29T10:47:20.426 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:20.443+0000 7f5729263640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:47:20.426 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:20.443+0000 7f5729263640 10 monclient(hunting): picked mon.noname-b con 0x7f57240b0860 addr v1:172.21.15.88:6790/0 2024-03-29T10:47:20.426 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:20.443+0000 7f5722ffd640 1 --1- >> v1:172.21.15.88:6790/0 conn(0x7f57240b0860 0x7f572413e820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6790/0 says I am v1:172.21.15.88:39244/0 (socket says 172.21.15.88:39244) 2024-03-29T10:47:20.426 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:20.443+0000 7f5722ffd640 1 -- 172.21.15.88:0/1766925964 learned_addr learned my addr 172.21.15.88:0/1766925964 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-29T10:47:20.426 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:20.443+0000 7f5729263640 10 monclient(hunting): picked mon.noname-a con 0x7f572413ef30 addr v1:172.21.15.88:6789/0 2024-03-29T10:47:20.427 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:20.443+0000 7f5729263640 10 monclient(hunting): picked mon.noname-c con 0x7f57240af8e0 addr v1:172.21.15.196:6789/0 2024-03-29T10:47:20.427 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:20.443+0000 7f5729263640 1 -- 172.21.15.88:0/1766925964 --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57240b0c40 con 0x7f572413ef30 2024-03-29T10:47:20.427 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:20.443+0000 7f5729263640 1 -- 172.21.15.88:0/1766925964 --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57240aed10 con 0x7f57240af8e0 2024-03-29T10:47:20.427 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:20.443+0000 7f5729263640 1 -- 172.21.15.88:0/1766925964 --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57240afd50 con 0x7f57240b0860 2024-03-29T10:47:20.427 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:20.443+0000 7f5729263640 10 monclient(hunting): _renew_subs 2024-03-29T10:47:20.427 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:20.443+0000 7f5729263640 10 monclient(hunting): authenticate will time out at 1162.679617s 2024-03-29T10:47:20.427 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:20.444+0000 7f5721ffb640 1 -- 172.21.15.88:0/1766925964 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2846345757 0 0) 0x7f57240afd50 con 0x7f57240b0860 2024-03-29T10:47:20.427 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:20.444+0000 7f5721ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:47:20.427 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:20.444+0000 7f5721ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:20.428 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:20.444+0000 7f5721ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:20.428 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:20.444+0000 7f5721ffb640 10 monclient(hunting): my global_id is 4496 2024-03-29T10:47:20.428 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:20.444+0000 7f5721ffb640 1 -- 172.21.15.88:0/1766925964 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f570c003440 con 0x7f57240b0860 2024-03-29T10:47:20.429 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:20.444+0000 7f5721ffb640 1 -- 172.21.15.88:0/1766925964 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2130471209 0 0) 0x7f570c003440 con 0x7f57240b0860 2024-03-29T10:47:20.430 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:20.444+0000 7f5721ffb640 1 -- 172.21.15.88:0/1766925964 >> v1:172.21.15.196:6789/0 conn(0x7f57240af8e0 legacy=0x7f5724149a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:20.430 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:20.444+0000 7f5721ffb640 1 -- 172.21.15.88:0/1766925964 >> v1:172.21.15.88:6789/0 conn(0x7f572413ef30 legacy=0x7f57240af1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:20.430 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:20.444+0000 7f5721ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:47:20.430 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:20.444+0000 7f5721ffb640 1 monclient: found mon.noname-b 2024-03-29T10:47:20.430 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:20.444+0000 7f5721ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:47:20.430 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:20.444+0000 7f5721ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.88:6790/0 2024-03-29T10:47:20.430 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:20.444+0000 7f5721ffb640 1 -- 172.21.15.88:0/1766925964 --> v1:172.21.15.88:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57240ab1b0 con 0x7f57240b0860 2024-03-29T10:47:20.430 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:20.444+0000 7f5721ffb640 10 monclient: _finish_auth 0 2024-03-29T10:47:20.430 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:20.444+0000 7f5721ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:47:20.430 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:20.444+0000 7f5721ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:47:20.430 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:20.444+0000 7f5721ffb640 1 -- 172.21.15.88:0/1766925964 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f571c001690 con 0x7f57240b0860 2024-03-29T10:47:20.430 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:20.444+0000 7f5721ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:20.430 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:20.444+0000 7f5721ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:47:20.431 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:20.444+0000 7f5721ffb640 10 monclient: dump: 2024-03-29T10:47:20.431 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:47:20.431 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:47:20.431 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:47:20.431 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:47:20.431 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:20.431 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:47:20.431 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:47:20.431 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:47:20.431 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:47:20.431 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:47:20.431 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:20.444+0000 7f5721ffb640 1 -- 172.21.15.88:0/1766925964 <== mon.2 v1:172.21.15.88:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f571c0018b0 con 0x7f57240b0860 2024-03-29T10:47:20.431 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:20.444+0000 7f5721ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:47:20.431 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:20.444+0000 7f5721ffb640 1 -- 172.21.15.88:0/1766925964 <== mon.2 v1:172.21.15.88:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f571c001c40 con 0x7f57240b0860 2024-03-29T10:47:20.432 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:20.444+0000 7f5721ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:20.432 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:20.444+0000 7f5721ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:47:20.432 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:20.444+0000 7f5721ffb640 10 monclient: dump: 2024-03-29T10:47:20.432 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:47:20.432 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:47:20.432 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:47:20.432 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:47:20.432 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:20.432 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:47:20.432 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:47:20.432 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:47:20.432 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:47:20.432 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:47:20.432 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:20.444+0000 7f5729263640 5 monclient: authenticate success, global_id 4496 2024-03-29T10:47:20.433 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:20.444+0000 7f5729263640 10 monclient: get_monmap_and_config success 2024-03-29T10:47:20.433 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:20.444+0000 7f5729263640 4 set_mon_vals no callback set 2024-03-29T10:47:20.433 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:20.445+0000 7f5729263640 10 monclient: shutdown 2024-03-29T10:47:20.433 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:20.445+0000 7f5729263640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:47:20.433 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:20.445+0000 7f5729263640 1 -- 172.21.15.88:0/1766925964 >> v1:172.21.15.88:6790/0 conn(0x7f57240b0860 legacy=0x7f572413e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:20.433 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:20.445+0000 7f5729263640 1 -- 172.21.15.88:0/1766925964 shutdown_connections 2024-03-29T10:47:20.433 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:20.445+0000 7f5729263640 1 -- 172.21.15.88:0/1766925964 >> 172.21.15.88:0/1766925964 conn(0x7f572406b660 msgr2=0x7f572406ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:47:20.433 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:20.446+0000 7f5729263640 1 -- 172.21.15.88:0/1766925964 shutdown_connections 2024-03-29T10:47:20.433 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:20.446+0000 7f5729263640 1 -- 172.21.15.88:0/1766925964 wait complete. 2024-03-29T10:47:20.433 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:20.447+0000 7f5729263640 10 monclient: build_initial_monmap 2024-03-29T10:47:20.433 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:20.447+0000 7f5729263640 10 monclient: monmap: 2024-03-29T10:47:20.433 INFO:teuthology.misc.health.smithi088.stderr:epoch 0 2024-03-29T10:47:20.433 INFO:teuthology.misc.health.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:47:20.433 INFO:teuthology.misc.health.smithi088.stderr:last_changed 0.000000 2024-03-29T10:47:20.433 INFO:teuthology.misc.health.smithi088.stderr:created 0.000000 2024-03-29T10:47:20.434 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 0 (unknown) 2024-03-29T10:47:20.434 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:47:20.434 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-03-29T10:47:20.434 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-29T10:47:20.434 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-c 2024-03-29T10:47:20.434 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:47:20.434 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:20.447+0000 7f5729263640 1 librados: starting msgr at 2024-03-29T10:47:20.434 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:20.447+0000 7f5729263640 1 librados: starting objecter 2024-03-29T10:47:20.437 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:20.448+0000 7f5729263640 1 Processor -- start 2024-03-29T10:47:20.438 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:20.448+0000 7f5729263640 1 -- start start 2024-03-29T10:47:20.438 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:20.448+0000 7f5729263640 1 librados: setting wanted keys 2024-03-29T10:47:20.438 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:20.448+0000 7f5729263640 1 librados: calling monclient init 2024-03-29T10:47:20.438 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:20.448+0000 7f5729263640 10 monclient: init 2024-03-29T10:47:20.438 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:20.449+0000 7f5729263640 10 monclient: _reopen_session rank -1 2024-03-29T10:47:20.438 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:20.449+0000 7f5729263640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:47:20.438 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:20.449+0000 7f5729263640 10 monclient(hunting): picked mon.noname-b con 0x7f57240af8e0 addr v1:172.21.15.196:6789/0 2024-03-29T10:47:20.438 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:20.449+0000 7f5729263640 10 monclient(hunting): picked mon.noname-c con 0x7f57240b0860 addr v1:172.21.15.88:6790/0 2024-03-29T10:47:20.438 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:20.449+0000 7f5729263640 10 monclient(hunting): picked mon.noname-a con 0x7f572413ef30 addr v1:172.21.15.88:6789/0 2024-03-29T10:47:20.438 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:20.449+0000 7f5729263640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57240b4050 con 0x7f572413ef30 2024-03-29T10:47:20.438 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:20.449+0000 7f5729263640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57241d3460 con 0x7f57240af8e0 2024-03-29T10:47:20.438 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:20.449+0000 7f5729263640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57240b3110 con 0x7f57240b0860 2024-03-29T10:47:20.439 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:20.449+0000 7f5729263640 10 monclient(hunting): _renew_subs 2024-03-29T10:47:20.439 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:20.449+0000 7f5729263640 10 monclient(hunting): authenticate will time out at 1162.685339s 2024-03-29T10:47:20.439 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:20.449+0000 7f57237fe640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f572413ef30 0x7f57241326f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:51516/0 (socket says 172.21.15.88:51516) 2024-03-29T10:47:20.439 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:20.449+0000 7f57237fe640 1 -- 172.21.15.88:0/1598834160 learned_addr learned my addr 172.21.15.88:0/1598834160 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-29T10:47:20.439 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:20.449+0000 7f570b7fe640 10 client.?.objecter ms_handle_connect 0x7f572413ef30 2024-03-29T10:47:20.439 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:20.449+0000 7f570b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:47:20.439 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:20.449+0000 7f570b7fe640 1 -- 172.21.15.88:0/1598834160 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 359234301 0 0) 0x7f57240b4050 con 0x7f572413ef30 2024-03-29T10:47:20.439 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:20.449+0000 7f570b7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:47:20.439 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:20.449+0000 7f570b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:20.439 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:20.449+0000 7f570b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:20.439 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:20.449+0000 7f570b7fe640 10 monclient(hunting): my global_id is 4470 2024-03-29T10:47:20.439 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:20.449+0000 7f570b7fe640 1 -- 172.21.15.88:0/1598834160 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5710004eb0 con 0x7f572413ef30 2024-03-29T10:47:20.439 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:20.450+0000 7f570b7fe640 10 client.?.objecter ms_handle_connect 0x7f57240b0860 2024-03-29T10:47:20.439 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:20.450+0000 7f570b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:47:20.440 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:20.450+0000 7f570b7fe640 1 -- 172.21.15.88:0/1598834160 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3495677630 0 0) 0x7f5710004eb0 con 0x7f572413ef30 2024-03-29T10:47:20.440 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:20.450+0000 7f570b7fe640 1 -- 172.21.15.88:0/1598834160 --> v1:172.21.15.88:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5710005300 con 0x7f572413ef30 2024-03-29T10:47:20.440 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:20.450+0000 7f570b7fe640 1 -- 172.21.15.88:0/1598834160 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f5714001cf0 con 0x7f572413ef30 2024-03-29T10:47:20.440 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:20.450+0000 7f570b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:20.440 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:20.450+0000 7f570b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:47:20.440 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:20.450+0000 7f570b7fe640 10 monclient(hunting): dump: 2024-03-29T10:47:20.440 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:47:20.440 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:47:20.440 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:47:20.440 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:47:20.440 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:20.440 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:47:20.440 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:47:20.440 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:47:20.441 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:47:20.441 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:47:20.441 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:20.450+0000 7f570b7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:47:20.441 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:20.450+0000 7f570b7fe640 1 -- 172.21.15.88:0/1598834160 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 662585157 0 0) 0x7f57240b3110 con 0x7f57240b0860 2024-03-29T10:47:20.441 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:20.450+0000 7f570b7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:47:20.441 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:20.450+0000 7f570b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:20.441 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:20.450+0000 7f570b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:20.441 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:20.450+0000 7f570b7fe640 10 monclient(hunting): my global_id is 4499 2024-03-29T10:47:20.441 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:20.450+0000 7f570b7fe640 1 -- 172.21.15.88:0/1598834160 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5710004eb0 con 0x7f57240b0860 2024-03-29T10:47:20.441 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:20.450+0000 7f570b7fe640 1 -- 172.21.15.88:0/1598834160 <== mon.0 v1:172.21.15.88:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 519678176 0 0) 0x7f5710005300 con 0x7f572413ef30 2024-03-29T10:47:20.441 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:20.450+0000 7f570b7fe640 1 -- 172.21.15.88:0/1598834160 >> v1:172.21.15.88:6790/0 conn(0x7f57240b0860 legacy=0x7f57241d3980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:20.441 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:20.450+0000 7f570b7fe640 1 -- 172.21.15.88:0/1598834160 >> v1:172.21.15.196:6789/0 conn(0x7f57240af8e0 legacy=0x7f57240b2a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:20.441 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:20.450+0000 7f570b7fe640 10 monclient: _finish_hunting 0 2024-03-29T10:47:20.441 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:20.450+0000 7f570b7fe640 1 monclient: found mon.a 2024-03-29T10:47:20.442 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:20.450+0000 7f570b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:47:20.442 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:20.450+0000 7f570b7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-03-29T10:47:20.442 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:20.450+0000 7f570b7fe640 1 -- 172.21.15.88:0/1598834160 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57240b32a0 con 0x7f572413ef30 2024-03-29T10:47:20.442 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:20.450+0000 7f570b7fe640 10 monclient: _finish_auth 0 2024-03-29T10:47:20.442 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:20.450+0000 7f570b7fe640 10 monclient: _check_auth_tickets 2024-03-29T10:47:20.442 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:20.450+0000 7f570b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:47:20.442 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:20.450+0000 7f5729263640 5 monclient: authenticate success, global_id 4470 2024-03-29T10:47:20.442 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:20.450+0000 7f5729263640 10 monclient: _renew_subs 2024-03-29T10:47:20.442 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:20.450+0000 7f5729263640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-03-29T10:47:20.442 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:20.450+0000 7f5729263640 1 -- 172.21.15.88:0/1598834160 --> v1:172.21.15.88:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57241d4090 con 0x7f572413ef30 2024-03-29T10:47:20.442 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:20.450+0000 7f5729263640 10 client.4470.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:47:20.442 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:20.450+0000 7f5729263640 10 monclient: _renew_subs 2024-03-29T10:47:20.442 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:20.450+0000 7f5729263640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-03-29T10:47:20.442 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:20.450+0000 7f5729263640 1 -- 172.21.15.88:0/1598834160 --> v1:172.21.15.88:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f572413a360 con 0x7f572413ef30 2024-03-29T10:47:20.443 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:20.450+0000 7f5729263640 1 librados: init done 2024-03-29T10:47:20.443 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:20.451+0000 7f570b7fe640 1 -- 172.21.15.88:0/1598834160 <== mon.0 v1:172.21.15.88:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5714003990 con 0x7f572413ef30 2024-03-29T10:47:20.443 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:20.451+0000 7f570b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:47:20.443 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:20.451+0000 7f570b7fe640 1 -- 172.21.15.88:0/1598834160 <== mon.0 v1:172.21.15.88:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f5714003da0 con 0x7f572413ef30 2024-03-29T10:47:20.443 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:20.451+0000 7f570b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:20.443 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:20.451+0000 7f570b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:47:20.443 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:20.451+0000 7f57217fa640 4 set_mon_vals no callback set 2024-03-29T10:47:20.443 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:20.451+0000 7f570b7fe640 10 monclient: dump: 2024-03-29T10:47:20.443 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:47:20.443 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:47:20.443 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:47:20.443 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:47:20.443 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:20.443 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:47:20.443 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:47:20.444 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:47:20.444 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:47:20.444 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:47:20.444 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:20.453+0000 7f570b7fe640 1 -- 172.21.15.88:0/1598834160 <== mon.0 v1:172.21.15.88:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4235629220 0 0) 0x7f5714003f80 con 0x7f572413ef30 2024-03-29T10:47:20.444 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:20.453+0000 7f5729263640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:20.444 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:20.453+0000 7f5729263640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:20.444 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:20.454+0000 7f5729263640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-03-29T10:47:20.444 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:20.454+0000 7f5729263640 1 -- 172.21.15.88:0/1598834160 --> v1:172.21.15.88:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5704005080 con 0x7f572413ef30 2024-03-29T10:47:20.444 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:20.454+0000 7f570b7fe640 1 -- 172.21.15.88:0/1598834160 <== mon.0 v1:172.21.15.88:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4257673582 0 0) 0x7f5714084e10 con 0x7f572413ef30 2024-03-29T10:47:20.444 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:20.454+0000 7f570b7fe640 10 client.4470.objecter ms_dispatch 0x7f5724000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:47:20.444 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:20.454+0000 7f570b7fe640 3 client.4470.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:47:20.444 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:20.454+0000 7f570b7fe640 3 client.4470.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:47:20.444 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:20.454+0000 7f570b7fe640 20 client.4470.objecter dump_active .. 0 homeless 2024-03-29T10:47:20.444 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:20.454+0000 7f5722ffd640 10 monclient: get_auth_request con 0x7f571006de80 auth_method 0 2024-03-29T10:47:20.445 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:20.455+0000 7f570b7fe640 10 client.4470.objecter ms_handle_connect 0x7f571006de80 2024-03-29T10:47:20.445 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:20.462+0000 7f570b7fe640 1 -- 172.21.15.88:0/1598834160 <== mon.0 v1:172.21.15.88:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5714050750 con 0x7f572413ef30 2024-03-29T10:47:20.445 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:20.462+0000 7f570b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:20.445 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:20.462+0000 7f570b7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:47:20.810 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:20.828+0000 7f5729263640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:47:20.811 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:20.828+0000 7f5729263640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:47:20.811 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:20.828+0000 7f5729263640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-03-29T10:47:20.811 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:20.828+0000 7f5729263640 1 -- 172.21.15.88:0/1598834160 --> v1:172.21.15.88:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5704005330 con 0x7f572413ef30 2024-03-29T10:47:20.811 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:20.828+0000 7f570b7fe640 1 -- 172.21.15.88:0/1598834160 <== mon.0 v1:172.21.15.88:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 344506374) 0x7f57140844c0 con 0x7f572413ef30 2024-03-29T10:47:20.812 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:20.828+0000 7f570b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:47:20.812 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:20.828+0000 7f570b7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:47:20.812 INFO:teuthology.misc.health.smithi088.stdout: 2024-03-29T10:47:20.812 INFO:teuthology.misc.health.smithi088.stdout:{"epoch":10,"fsid":"869f5188-f92b-42dd-a46d-8c646e0388a9","created":"2024-03-29T10:40:40.285087+0000","modified":"2024-03-29T10:40:42.536230+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:40:42.536230+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a26c0c1c-8de0-438c-a3f6-3df10ea61574","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5de24915-8dec-49b6-8b16-f3170197c591","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ac054af3-d589-4565-9644-e6af75975c53","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3ec4f045-c714-4a22-bd1a-e07330b45ab3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"cddb9867-8e30-4951-9b13-3d1e78b9c45a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dc6d9911-785d-432f-8b53-56a1ccab302f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"77c88437-b587-496e-8b4d-1824ea8b5305","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ce963d96-5a8e-44c3-99de-c0dbebe234ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:47:20.816 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:20.833+0000 7f5729263640 10 librados: watch_flush enter 2024-03-29T10:47:20.816 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:20.833+0000 7f5729263640 10 librados: watch_flush exit 2024-03-29T10:47:20.816 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:20.833+0000 7f5729263640 20 client.4470.objecter shutdown clearing up homeless session... 2024-03-29T10:47:20.816 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:20.833+0000 7f5729263640 10 client.4470.objecter successfully canceled tick 2024-03-29T10:47:20.817 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:20.833+0000 7f5729263640 1 -- 172.21.15.88:0/1598834160 >> v1:172.21.15.88:6816/1752861349 conn(0x7f571006de80 legacy=0x7f5710070340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:20.817 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:20.833+0000 7f5729263640 10 monclient: shutdown 2024-03-29T10:47:20.817 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:20.833+0000 7f5729263640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:47:20.817 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:20.833+0000 7f5729263640 1 -- 172.21.15.88:0/1598834160 >> v1:172.21.15.88:6789/0 conn(0x7f572413ef30 legacy=0x7f57241326f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:20.817 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:20.833+0000 7f5729263640 1 -- 172.21.15.88:0/1598834160 shutdown_connections 2024-03-29T10:47:20.817 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:20.833+0000 7f5729263640 1 -- 172.21.15.88:0/1598834160 >> 172.21.15.88:0/1598834160 conn(0x7f572406b660 msgr2=0x7f572406ba40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:47:20.817 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:20.833+0000 7f5729263640 1 -- 172.21.15.88:0/1598834160 shutdown_connections 2024-03-29T10:47:20.817 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:20.833+0000 7f5729263640 1 -- 172.21.15.88:0/1598834160 wait complete. 2024-03-29T10:47:20.817 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:20.834+0000 7f5729263640 1 librados: shutdown 2024-03-29T10:47:20.830 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:47:27.638 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:47:27.772 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:27.789+0000 7f5dd289d640 10 monclient: get_monmap_and_config 2024-03-29T10:47:27.772 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:27.789+0000 7f5dd289d640 10 monclient: build_initial_monmap 2024-03-29T10:47:27.772 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:27.789+0000 7f5dd289d640 10 monclient: monmap: 2024-03-29T10:47:27.772 INFO:teuthology.misc.health.smithi088.stderr:epoch 0 2024-03-29T10:47:27.773 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:47:27.773 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:47:27.790955+0000 2024-03-29T10:47:27.773 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:47:27.790955+0000 2024-03-29T10:47:27.773 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 0 (unknown) 2024-03-29T10:47:27.773 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:47:27.773 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-03-29T10:47:27.773 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-29T10:47:27.773 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-b 2024-03-29T10:47:27.773 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:47:27.774 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:27.791+0000 7f5dd289d640 1 Processor -- start 2024-03-29T10:47:27.774 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:27.791+0000 7f5dd289d640 1 -- start start 2024-03-29T10:47:27.774 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:27.791+0000 7f5dd289d640 10 monclient: init 2024-03-29T10:47:27.775 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:27.792+0000 7f5dd289d640 10 monclient: _reopen_session rank -1 2024-03-29T10:47:27.775 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:27.792+0000 7f5dd289d640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:47:27.775 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:27.792+0000 7f5dd289d640 10 monclient(hunting): picked mon.noname-b con 0x7f5dcc1457f0 addr v1:172.21.15.88:6790/0 2024-03-29T10:47:27.775 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:27.792+0000 7f5dd289d640 10 monclient(hunting): picked mon.noname-a con 0x7f5dcc13f650 addr v1:172.21.15.88:6789/0 2024-03-29T10:47:27.775 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:27.792+0000 7f5dd289d640 10 monclient(hunting): picked mon.noname-c con 0x7f5dcc0a9d70 addr v1:172.21.15.196:6789/0 2024-03-29T10:47:27.775 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:27.792+0000 7f5dd289d640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5dcc0aaef0 con 0x7f5dcc13f650 2024-03-29T10:47:27.776 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:27.792+0000 7f5dd289d640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5dcc145bd0 con 0x7f5dcc0a9d70 2024-03-29T10:47:27.776 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:27.792+0000 7f5dd289d640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5dcc145eb0 con 0x7f5dcc1457f0 2024-03-29T10:47:27.776 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:27.792+0000 7f5dd289d640 10 monclient(hunting): _renew_subs 2024-03-29T10:47:27.776 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:27.792+0000 7f5dd289d640 10 monclient(hunting): authenticate will time out at 1170.028408s 2024-03-29T10:47:27.776 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:27.792+0000 7f5dcbfff640 1 --1- >> v1:172.21.15.88:6790/0 conn(0x7f5dcc1457f0 0x7f5dcc13ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6790/0 says I am v1:172.21.15.88:47588/0 (socket says 172.21.15.88:47588) 2024-03-29T10:47:27.776 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:27.792+0000 7f5dcbfff640 1 -- 172.21.15.88:0/1967624454 learned_addr learned my addr 172.21.15.88:0/1967624454 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-29T10:47:27.776 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:27.793+0000 7f5dcaffd640 1 -- 172.21.15.88:0/1967624454 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 982468435 0 0) 0x7f5dcc145eb0 con 0x7f5dcc1457f0 2024-03-29T10:47:27.776 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:27.793+0000 7f5dcaffd640 10 monclient(hunting): _negotiate 2024-03-29T10:47:27.776 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:27.793+0000 7f5dcaffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:27.776 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:27.793+0000 7f5dcaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:27.776 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:27.793+0000 7f5dcaffd640 10 monclient(hunting): my global_id is 4502 2024-03-29T10:47:27.776 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:27.793+0000 7f5dcaffd640 1 -- 172.21.15.88:0/1967624454 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5db8003520 con 0x7f5dcc1457f0 2024-03-29T10:47:27.778 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:27.793+0000 7f5dcaffd640 1 -- 172.21.15.88:0/1967624454 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 62790043 0 0) 0x7f5dcc145bd0 con 0x7f5dcc0a9d70 2024-03-29T10:47:27.778 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:27.793+0000 7f5dcaffd640 10 monclient(hunting): _negotiate 2024-03-29T10:47:27.778 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:27.793+0000 7f5dcaffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:27.778 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:27.793+0000 7f5dcaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:27.778 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:27.793+0000 7f5dcaffd640 10 monclient(hunting): my global_id is 4459 2024-03-29T10:47:27.778 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:27.793+0000 7f5dcaffd640 1 -- 172.21.15.88:0/1967624454 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5dcc145eb0 con 0x7f5dcc0a9d70 2024-03-29T10:47:27.778 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:27.794+0000 7f5dcaffd640 1 -- 172.21.15.88:0/1967624454 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 717370407 0 0) 0x7f5db8003520 con 0x7f5dcc1457f0 2024-03-29T10:47:27.778 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:27.794+0000 7f5dcaffd640 1 -- 172.21.15.88:0/1967624454 >> v1:172.21.15.196:6789/0 conn(0x7f5dcc0a9d70 legacy=0x7f5dcc14d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:27.779 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:27.794+0000 7f5dcaffd640 1 -- 172.21.15.88:0/1967624454 >> v1:172.21.15.88:6789/0 conn(0x7f5dcc13f650 legacy=0x7f5dcc0b2620 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:47:27.779 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:27.794+0000 7f5dcaffd640 10 monclient: _finish_hunting 0 2024-03-29T10:47:27.779 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:27.794+0000 7f5dcaffd640 1 monclient: found mon.noname-b 2024-03-29T10:47:27.779 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:27.794+0000 7f5dcaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:47:27.779 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:27.794+0000 7f5dcaffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.88:6790/0 2024-03-29T10:47:27.779 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:27.794+0000 7f5dcaffd640 1 -- 172.21.15.88:0/1967624454 --> v1:172.21.15.88:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5dcc146340 con 0x7f5dcc1457f0 2024-03-29T10:47:27.779 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:27.794+0000 7f5dcaffd640 10 monclient: _finish_auth 0 2024-03-29T10:47:27.779 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:27.794+0000 7f5dcaffd640 10 monclient: _check_auth_tickets 2024-03-29T10:47:27.779 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:27.794+0000 7f5dcaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:47:27.779 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:27.795+0000 7f5dcaffd640 1 -- 172.21.15.88:0/1967624454 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f5db4001690 con 0x7f5dcc1457f0 2024-03-29T10:47:27.780 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:27.795+0000 7f5dcaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:27.780 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:27.795+0000 7f5dcaffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:47:27.780 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:27.795+0000 7f5dcaffd640 10 monclient: dump: 2024-03-29T10:47:27.780 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:47:27.780 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:47:27.780 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:47:27.780 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:47:27.780 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:27.780 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:47:27.780 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:47:27.780 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:47:27.780 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:47:27.781 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:47:27.781 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:27.795+0000 7f5dd289d640 5 monclient: authenticate success, global_id 4502 2024-03-29T10:47:27.781 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:27.795+0000 7f5dd289d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:47:27.781 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:27.795+0000 7f5dcaffd640 1 -- 172.21.15.88:0/1967624454 <== mon.2 v1:172.21.15.88:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5db40017f0 con 0x7f5dcc1457f0 2024-03-29T10:47:27.781 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:27.795+0000 7f5dcaffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:47:27.781 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:27.795+0000 7f5dcaffd640 1 -- 172.21.15.88:0/1967624454 <== mon.2 v1:172.21.15.88:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f5db4001c00 con 0x7f5dcc1457f0 2024-03-29T10:47:27.781 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:27.795+0000 7f5dcaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:27.781 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:27.795+0000 7f5dcaffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:47:27.781 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:27.795+0000 7f5dcaffd640 10 monclient: dump: 2024-03-29T10:47:27.781 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:47:27.781 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:47:27.781 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:47:27.781 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:47:27.782 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:27.782 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:47:27.782 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:47:27.782 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:47:27.782 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:47:27.782 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:47:27.782 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:27.795+0000 7f5dd289d640 10 monclient: get_monmap_and_config success 2024-03-29T10:47:27.782 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:27.795+0000 7f5dd289d640 4 set_mon_vals no callback set 2024-03-29T10:47:27.785 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:27.796+0000 7f5dd289d640 10 monclient: shutdown 2024-03-29T10:47:27.786 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:27.796+0000 7f5dd289d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:47:27.786 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:27.796+0000 7f5dd289d640 1 -- 172.21.15.88:0/1967624454 >> v1:172.21.15.88:6790/0 conn(0x7f5dcc1457f0 legacy=0x7f5dcc13ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:27.786 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:27.796+0000 7f5dd289d640 1 -- 172.21.15.88:0/1967624454 shutdown_connections 2024-03-29T10:47:27.786 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:27.796+0000 7f5dd289d640 1 -- 172.21.15.88:0/1967624454 >> 172.21.15.88:0/1967624454 conn(0x7f5dcc066a00 msgr2=0x7f5dcc066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:47:27.786 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:27.796+0000 7f5dd289d640 1 -- 172.21.15.88:0/1967624454 shutdown_connections 2024-03-29T10:47:27.786 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:27.796+0000 7f5dd289d640 1 -- 172.21.15.88:0/1967624454 wait complete. 2024-03-29T10:47:27.786 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:27.797+0000 7f5dd289d640 10 monclient: build_initial_monmap 2024-03-29T10:47:27.786 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:27.797+0000 7f5dd289d640 10 monclient: monmap: 2024-03-29T10:47:27.787 INFO:teuthology.misc.health.smithi088.stderr:epoch 0 2024-03-29T10:47:27.787 INFO:teuthology.misc.health.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:47:27.787 INFO:teuthology.misc.health.smithi088.stderr:last_changed 0.000000 2024-03-29T10:47:27.787 INFO:teuthology.misc.health.smithi088.stderr:created 0.000000 2024-03-29T10:47:27.787 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 0 (unknown) 2024-03-29T10:47:27.787 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:47:27.787 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-03-29T10:47:27.787 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-29T10:47:27.787 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-c 2024-03-29T10:47:27.787 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:47:27.787 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:27.798+0000 7f5dd289d640 1 librados: starting msgr at 2024-03-29T10:47:27.788 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:27.798+0000 7f5dd289d640 1 librados: starting objecter 2024-03-29T10:47:27.788 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:27.798+0000 7f5dd289d640 1 Processor -- start 2024-03-29T10:47:27.788 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:27.798+0000 7f5dd289d640 1 -- start start 2024-03-29T10:47:27.788 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:27.798+0000 7f5dd289d640 1 librados: setting wanted keys 2024-03-29T10:47:27.788 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:27.798+0000 7f5dd289d640 1 librados: calling monclient init 2024-03-29T10:47:27.788 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:27.798+0000 7f5dd289d640 10 monclient: init 2024-03-29T10:47:27.788 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:27.799+0000 7f5dd289d640 10 monclient: _reopen_session rank -1 2024-03-29T10:47:27.788 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:27.799+0000 7f5dd289d640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:47:27.788 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:27.799+0000 7f5dd289d640 10 monclient(hunting): picked mon.noname-b con 0x7f5dcc0a9d70 addr v1:172.21.15.196:6789/0 2024-03-29T10:47:27.788 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:27.799+0000 7f5dd289d640 10 monclient(hunting): picked mon.noname-c con 0x7f5dcc13f650 addr v1:172.21.15.88:6790/0 2024-03-29T10:47:27.788 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:27.799+0000 7f5dd289d640 10 monclient(hunting): picked mon.noname-a con 0x7f5dcc1457f0 addr v1:172.21.15.88:6789/0 2024-03-29T10:47:27.788 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:27.799+0000 7f5dd289d640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5dcc139b30 con 0x7f5dcc1457f0 2024-03-29T10:47:27.788 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:27.799+0000 7f5dd289d640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5dcc13d2a0 con 0x7f5dcc0a9d70 2024-03-29T10:47:27.789 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:27.799+0000 7f5dd289d640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5dcc13c9d0 con 0x7f5dcc13f650 2024-03-29T10:47:27.789 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:27.799+0000 7f5dcb7fe640 1 --1- >> v1:172.21.15.88:6790/0 conn(0x7f5dcc13f650 0x7f5dcc1df800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6790/0 says I am v1:172.21.15.88:47890/0 (socket says 172.21.15.88:47890) 2024-03-29T10:47:27.789 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:27.799+0000 7f5dcb7fe640 1 -- 172.21.15.88:0/2125717622 learned_addr learned my addr 172.21.15.88:0/2125717622 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-29T10:47:27.789 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:27.799+0000 7f5dd289d640 10 monclient(hunting): _renew_subs 2024-03-29T10:47:27.789 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:27.799+0000 7f5dd289d640 10 monclient(hunting): authenticate will time out at 1170.035453s 2024-03-29T10:47:27.789 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:27.799+0000 7f5dc8ff9640 10 client.?.objecter ms_handle_connect 0x7f5dcc13f650 2024-03-29T10:47:27.789 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:27.799+0000 7f5dc8ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:47:27.789 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:27.799+0000 7f5dc8ff9640 1 -- 172.21.15.88:0/2125717622 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2607217120 0 0) 0x7f5dcc13c9d0 con 0x7f5dcc13f650 2024-03-29T10:47:27.789 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:27.800+0000 7f5dc8ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:47:27.789 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:27.800+0000 7f5dc8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:27.789 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:27.800+0000 7f5dc8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:27.789 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:27.800+0000 7f5dc8ff9640 10 monclient(hunting): my global_id is 4505 2024-03-29T10:47:27.789 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:27.800+0000 7f5dc8ff9640 1 -- 172.21.15.88:0/2125717622 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5dac004eb0 con 0x7f5dcc13f650 2024-03-29T10:47:27.789 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:27.800+0000 7f5dc8ff9640 1 -- 172.21.15.88:0/2125717622 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4072562211 0 0) 0x7f5dac004eb0 con 0x7f5dcc13f650 2024-03-29T10:47:27.789 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:27.800+0000 7f5dc8ff9640 1 -- 172.21.15.88:0/2125717622 --> v1:172.21.15.88:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5dcc13c9d0 con 0x7f5dcc13f650 2024-03-29T10:47:27.790 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:27.800+0000 7f5dc8ff9640 1 -- 172.21.15.88:0/2125717622 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f5dbc001960 con 0x7f5dcc13f650 2024-03-29T10:47:27.790 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:27.800+0000 7f5dc8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:27.790 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:27.801+0000 7f5dc8ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:47:27.790 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:27.801+0000 7f5dc8ff9640 10 monclient(hunting): dump: 2024-03-29T10:47:27.790 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:47:27.790 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:47:27.790 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:47:27.790 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:47:27.790 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:27.790 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:47:27.790 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:47:27.790 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:47:27.790 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:47:27.790 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:47:27.791 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:27.801+0000 7f5dc8ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:47:27.791 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:27.801+0000 7f5dc8ff9640 1 -- 172.21.15.88:0/2125717622 <== mon.2 v1:172.21.15.88:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 116021939 0 0) 0x7f5dcc13c9d0 con 0x7f5dcc13f650 2024-03-29T10:47:27.791 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:27.801+0000 7f5dc8ff9640 1 -- 172.21.15.88:0/2125717622 >> v1:172.21.15.196:6789/0 conn(0x7f5dcc0a9d70 legacy=0x7f5dcc1df0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:27.791 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:27.801+0000 7f5dc8ff9640 1 -- 172.21.15.88:0/2125717622 >> v1:172.21.15.88:6789/0 conn(0x7f5dcc1457f0 legacy=0x7f5dcc1e63c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:47:27.791 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:27.803+0000 7f5dc8ff9640 10 monclient: _finish_hunting 0 2024-03-29T10:47:27.792 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:27.803+0000 7f5dc8ff9640 1 monclient: found mon.c 2024-03-29T10:47:27.792 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:27.803+0000 7f5dc8ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:47:27.792 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:27.803+0000 7f5dc8ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-03-29T10:47:27.792 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:27.803+0000 7f5dc8ff9640 1 -- 172.21.15.88:0/2125717622 --> v1:172.21.15.88:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5dcc13e6d0 con 0x7f5dcc13f650 2024-03-29T10:47:27.792 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:27.803+0000 7f5dc8ff9640 10 monclient: _finish_auth 0 2024-03-29T10:47:27.792 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:27.803+0000 7f5dc8ff9640 10 monclient: _check_auth_tickets 2024-03-29T10:47:27.793 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:27.803+0000 7f5dc8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:47:27.793 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:27.803+0000 7f5dd289d640 5 monclient: authenticate success, global_id 4505 2024-03-29T10:47:27.793 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:27.803+0000 7f5dd289d640 10 monclient: _renew_subs 2024-03-29T10:47:27.793 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:27.803+0000 7f5dd289d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-03-29T10:47:27.793 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:27.803+0000 7f5dd289d640 1 -- 172.21.15.88:0/2125717622 --> v1:172.21.15.88:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5dcc132ef0 con 0x7f5dcc13f650 2024-03-29T10:47:27.793 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:27.803+0000 7f5dc8ff9640 1 -- 172.21.15.88:0/2125717622 <== mon.2 v1:172.21.15.88:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5dbc001c80 con 0x7f5dcc13f650 2024-03-29T10:47:27.793 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:27.803+0000 7f5dc8ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:47:27.793 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:27.803+0000 7f5dc8ff9640 1 -- 172.21.15.88:0/2125717622 <== mon.2 v1:172.21.15.88:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f5dbc003a40 con 0x7f5dcc13f650 2024-03-29T10:47:27.793 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:27.803+0000 7f5dc8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:27.793 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:27.803+0000 7f5dc8ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:47:27.793 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:27.803+0000 7f5dc8ff9640 10 monclient: dump: 2024-03-29T10:47:27.793 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:47:27.794 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:47:27.794 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:47:27.794 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:47:27.794 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:27.794 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:47:27.794 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:47:27.794 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:47:27.794 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:47:27.794 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:47:27.794 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:27.803+0000 7f5dca7fc640 4 set_mon_vals no callback set 2024-03-29T10:47:27.794 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:27.804+0000 7f5dd289d640 10 client.4505.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:47:27.794 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:27.804+0000 7f5dd289d640 10 monclient: _renew_subs 2024-03-29T10:47:27.795 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:27.804+0000 7f5dd289d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-03-29T10:47:27.795 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:27.804+0000 7f5dd289d640 1 -- 172.21.15.88:0/2125717622 --> v1:172.21.15.88:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5dcc138f20 con 0x7f5dcc13f650 2024-03-29T10:47:27.795 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:27.804+0000 7f5dd289d640 1 librados: init done 2024-03-29T10:47:27.795 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:27.806+0000 7f5dc8ff9640 1 -- 172.21.15.88:0/2125717622 <== mon.2 v1:172.21.15.88:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4235629220 0 0) 0x7f5dbc003ba0 con 0x7f5dcc13f650 2024-03-29T10:47:27.795 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:27.807+0000 7f5dc8ff9640 1 -- 172.21.15.88:0/2125717622 <== mon.2 v1:172.21.15.88:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4257673582 0 0) 0x7f5dbc084b00 con 0x7f5dcc13f650 2024-03-29T10:47:27.795 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:27.807+0000 7f5dc8ff9640 10 client.4505.objecter ms_dispatch 0x7f5dcc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:47:27.795 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:27.807+0000 7f5dc8ff9640 3 client.4505.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:47:27.795 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:27.807+0000 7f5dc8ff9640 3 client.4505.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:47:27.795 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:27.807+0000 7f5dcbfff640 10 monclient: get_auth_request con 0x7f5dac06dc90 auth_method 0 2024-03-29T10:47:27.795 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:27.807+0000 7f5dc8ff9640 20 client.4505.objecter dump_active .. 0 homeless 2024-03-29T10:47:27.795 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:27.807+0000 7f5dc8ff9640 10 client.4505.objecter ms_handle_connect 0x7f5dac06dc90 2024-03-29T10:47:27.795 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:27.807+0000 7f5dd289d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:27.795 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:27.807+0000 7f5dd289d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:27.796 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:27.807+0000 7f5dd289d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-03-29T10:47:27.796 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:27.807+0000 7f5dd289d640 1 -- 172.21.15.88:0/2125717622 --> v1:172.21.15.88:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5dcc139850 con 0x7f5dcc13f650 2024-03-29T10:47:27.798 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:27.816+0000 7f5dc8ff9640 1 -- 172.21.15.88:0/2125717622 <== mon.2 v1:172.21.15.88:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5dcc139850 con 0x7f5dcc13f650 2024-03-29T10:47:27.798 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:27.816+0000 7f5dc8ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:27.798 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:27.816+0000 7f5dc8ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:47:28.106 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:28.124+0000 7f5dd289d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:47:28.106 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:28.124+0000 7f5dd289d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:47:28.107 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:28.124+0000 7f5dd289d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-03-29T10:47:28.107 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:28.124+0000 7f5dd289d640 1 -- 172.21.15.88:0/2125717622 --> v1:172.21.15.88:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5dcc139850 con 0x7f5dcc13f650 2024-03-29T10:47:28.107 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:28.125+0000 7f5dc8ff9640 1 -- 172.21.15.88:0/2125717622 <== mon.2 v1:172.21.15.88:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 344506374) 0x7f5dcc139850 con 0x7f5dcc13f650 2024-03-29T10:47:28.108 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:28.125+0000 7f5dc8ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:47:28.108 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:28.125+0000 7f5dc8ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:47:28.108 INFO:teuthology.misc.health.smithi088.stdout: 2024-03-29T10:47:28.108 INFO:teuthology.misc.health.smithi088.stdout:{"epoch":10,"fsid":"869f5188-f92b-42dd-a46d-8c646e0388a9","created":"2024-03-29T10:40:40.285087+0000","modified":"2024-03-29T10:40:42.536230+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:40:42.536230+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a26c0c1c-8de0-438c-a3f6-3df10ea61574","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5de24915-8dec-49b6-8b16-f3170197c591","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ac054af3-d589-4565-9644-e6af75975c53","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3ec4f045-c714-4a22-bd1a-e07330b45ab3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"cddb9867-8e30-4951-9b13-3d1e78b9c45a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dc6d9911-785d-432f-8b53-56a1ccab302f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"77c88437-b587-496e-8b4d-1824ea8b5305","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ce963d96-5a8e-44c3-99de-c0dbebe234ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:47:28.111 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:28.129+0000 7f5dd289d640 10 librados: watch_flush enter 2024-03-29T10:47:28.112 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:28.129+0000 7f5dd289d640 10 librados: watch_flush exit 2024-03-29T10:47:28.112 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:28.129+0000 7f5dd289d640 20 client.4505.objecter shutdown clearing up homeless session... 2024-03-29T10:47:28.112 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:28.129+0000 7f5dd289d640 10 client.4505.objecter successfully canceled tick 2024-03-29T10:47:28.112 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:28.129+0000 7f5dd289d640 1 -- 172.21.15.88:0/2125717622 >> v1:172.21.15.88:6816/1752861349 conn(0x7f5dac06dc90 legacy=0x7f5dac070150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:28.112 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:28.129+0000 7f5dd289d640 10 monclient: shutdown 2024-03-29T10:47:28.112 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:28.129+0000 7f5dd289d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:47:28.112 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:28.129+0000 7f5dd289d640 1 -- 172.21.15.88:0/2125717622 >> v1:172.21.15.88:6790/0 conn(0x7f5dcc13f650 legacy=0x7f5dcc1df800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:28.112 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:28.129+0000 7f5dd289d640 1 -- 172.21.15.88:0/2125717622 shutdown_connections 2024-03-29T10:47:28.112 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:28.129+0000 7f5dd289d640 1 -- 172.21.15.88:0/2125717622 >> 172.21.15.88:0/2125717622 conn(0x7f5dcc066a00 msgr2=0x7f5dcc0b4820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:47:28.112 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:28.129+0000 7f5dd289d640 1 -- 172.21.15.88:0/2125717622 shutdown_connections 2024-03-29T10:47:28.112 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:28.129+0000 7f5dd289d640 1 -- 172.21.15.88:0/2125717622 wait complete. 2024-03-29T10:47:28.112 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:28.129+0000 7f5dd289d640 1 librados: shutdown 2024-03-29T10:47:28.127 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:47:34.932 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:47:35.082 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:35.100+0000 7f032250e640 10 monclient: get_monmap_and_config 2024-03-29T10:47:35.083 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:35.100+0000 7f032250e640 10 monclient: build_initial_monmap 2024-03-29T10:47:35.083 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:35.100+0000 7f032250e640 10 monclient: monmap: 2024-03-29T10:47:35.083 INFO:teuthology.misc.health.smithi088.stderr:epoch 0 2024-03-29T10:47:35.083 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:47:35.083 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:47:35.101194+0000 2024-03-29T10:47:35.083 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:47:35.101194+0000 2024-03-29T10:47:35.083 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 0 (unknown) 2024-03-29T10:47:35.083 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:47:35.083 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-03-29T10:47:35.083 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-29T10:47:35.083 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-b 2024-03-29T10:47:35.083 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:47:35.086 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:35.104+0000 7f032250e640 1 Processor -- start 2024-03-29T10:47:35.087 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:35.104+0000 7f032250e640 1 -- start start 2024-03-29T10:47:35.087 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:35.104+0000 7f032250e640 10 monclient: init 2024-03-29T10:47:35.087 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:35.105+0000 7f032250e640 10 monclient: _reopen_session rank -1 2024-03-29T10:47:35.087 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:35.105+0000 7f032250e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:47:35.088 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:35.105+0000 7f032250e640 10 monclient(hunting): picked mon.noname-a con 0x7f031c0a9ac0 addr v1:172.21.15.88:6789/0 2024-03-29T10:47:35.088 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:35.105+0000 7f032250e640 10 monclient(hunting): picked mon.noname-b con 0x7f031c13ee90 addr v1:172.21.15.88:6790/0 2024-03-29T10:47:35.088 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:35.105+0000 7f032250e640 10 monclient(hunting): picked mon.noname-c con 0x7f031c0af900 addr v1:172.21.15.196:6789/0 2024-03-29T10:47:35.088 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:35.105+0000 7f032250e640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f031c0afd10 con 0x7f031c0a9ac0 2024-03-29T10:47:35.088 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:35.105+0000 7f032250e640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f031c0af6f0 con 0x7f031c0af900 2024-03-29T10:47:35.088 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:35.105+0000 7f032250e640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f031c0aac80 con 0x7f031c13ee90 2024-03-29T10:47:35.088 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:35.105+0000 7f032250e640 10 monclient(hunting): _renew_subs 2024-03-29T10:47:35.088 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:35.105+0000 7f032250e640 10 monclient(hunting): authenticate will time out at 1177.341049s 2024-03-29T10:47:35.096 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:35.110+0000 7f031bfff640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f031c0a9ac0 0x7f031c0adfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:47403/0 (socket says 172.21.15.88:47403) 2024-03-29T10:47:35.097 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:35.110+0000 7f031bfff640 1 -- 172.21.15.88:0/3109781761 learned_addr learned my addr 172.21.15.88:0/3109781761 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-29T10:47:35.097 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:35.112+0000 7f031affd640 1 -- 172.21.15.88:0/3109781761 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3107874874 0 0) 0x7f031c0afd10 con 0x7f031c0a9ac0 2024-03-29T10:47:35.097 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:35.112+0000 7f031affd640 10 monclient(hunting): _negotiate 2024-03-29T10:47:35.097 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:35.112+0000 7f031affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:35.097 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:35.112+0000 7f031affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:35.097 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:35.112+0000 7f031affd640 10 monclient(hunting): my global_id is 4473 2024-03-29T10:47:35.097 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:35.112+0000 7f031affd640 1 -- 172.21.15.88:0/3109781761 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0304003500 con 0x7f031c0a9ac0 2024-03-29T10:47:35.097 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:35.112+0000 7f031affd640 1 -- 172.21.15.88:0/3109781761 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 228333296 0 0) 0x7f031c0af6f0 con 0x7f031c0af900 2024-03-29T10:47:35.097 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:35.112+0000 7f031affd640 10 monclient(hunting): _negotiate 2024-03-29T10:47:35.097 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:35.112+0000 7f031affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:35.097 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:35.112+0000 7f031affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:35.097 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:35.112+0000 7f031affd640 10 monclient(hunting): my global_id is 4462 2024-03-29T10:47:35.097 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:35.112+0000 7f031affd640 1 -- 172.21.15.88:0/3109781761 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f031c0afd10 con 0x7f031c0af900 2024-03-29T10:47:35.098 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:35.112+0000 7f031affd640 1 -- 172.21.15.88:0/3109781761 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1728163319 0 0) 0x7f031c0aac80 con 0x7f031c13ee90 2024-03-29T10:47:35.098 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:35.112+0000 7f031affd640 10 monclient(hunting): _negotiate 2024-03-29T10:47:35.098 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:35.112+0000 7f031affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:35.098 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:35.112+0000 7f031affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:35.098 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:35.112+0000 7f031affd640 10 monclient(hunting): my global_id is 4508 2024-03-29T10:47:35.098 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:35.112+0000 7f031affd640 1 -- 172.21.15.88:0/3109781761 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f031c0af6f0 con 0x7f031c13ee90 2024-03-29T10:47:35.098 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:35.112+0000 7f031affd640 1 -- 172.21.15.88:0/3109781761 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 656806053 0 0) 0x7f0304003500 con 0x7f031c0a9ac0 2024-03-29T10:47:35.098 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:35.112+0000 7f031affd640 1 -- 172.21.15.88:0/3109781761 >> v1:172.21.15.88:6790/0 conn(0x7f031c13ee90 legacy=0x7f031c13b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:35.098 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:35.112+0000 7f031affd640 1 -- 172.21.15.88:0/3109781761 >> v1:172.21.15.196:6789/0 conn(0x7f031c0af900 legacy=0x7f031c149b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:35.098 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:35.112+0000 7f031affd640 10 monclient: _finish_hunting 0 2024-03-29T10:47:35.098 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:35.112+0000 7f031affd640 1 monclient: found mon.noname-a 2024-03-29T10:47:35.098 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:35.112+0000 7f031affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:47:35.098 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:35.112+0000 7f031affd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.88:6789/0 2024-03-29T10:47:35.098 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:35.112+0000 7f031affd640 1 -- 172.21.15.88:0/3109781761 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f031c0aaf60 con 0x7f031c0a9ac0 2024-03-29T10:47:35.099 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:35.112+0000 7f031affd640 10 monclient: _finish_auth 0 2024-03-29T10:47:35.099 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:35.112+0000 7f031affd640 10 monclient: _check_auth_tickets 2024-03-29T10:47:35.099 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:35.112+0000 7f031affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:47:35.099 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:35.112+0000 7f031affd640 1 -- 172.21.15.88:0/3109781761 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f0314004cb0 con 0x7f031c0a9ac0 2024-03-29T10:47:35.099 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:35.112+0000 7f031affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:35.099 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:35.112+0000 7f031affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:47:35.099 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:35.112+0000 7f031affd640 10 monclient: dump: 2024-03-29T10:47:35.099 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:47:35.099 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:47:35.099 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:47:35.099 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:47:35.099 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:35.099 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:47:35.100 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:47:35.100 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:47:35.100 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:47:35.100 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:47:35.100 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:35.112+0000 7f032250e640 5 monclient: authenticate success, global_id 4473 2024-03-29T10:47:35.100 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:35.112+0000 7f032250e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:47:35.100 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:35.114+0000 7f031affd640 1 -- 172.21.15.88:0/3109781761 <== mon.0 v1:172.21.15.88:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0314004e10 con 0x7f031c0a9ac0 2024-03-29T10:47:35.100 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:35.114+0000 7f031affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:47:35.100 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:35.114+0000 7f031affd640 1 -- 172.21.15.88:0/3109781761 <== mon.0 v1:172.21.15.88:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f03140013f0 con 0x7f031c0a9ac0 2024-03-29T10:47:35.100 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:35.114+0000 7f031affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:35.100 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:35.114+0000 7f031affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:47:35.100 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:35.114+0000 7f031affd640 10 monclient: dump: 2024-03-29T10:47:35.100 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:47:35.100 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:47:35.101 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:47:35.101 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:47:35.101 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:35.101 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:47:35.101 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:47:35.101 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:47:35.101 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:47:35.101 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:47:35.103 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:35.115+0000 7f032250e640 10 monclient: get_monmap_and_config success 2024-03-29T10:47:35.103 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:35.115+0000 7f032250e640 4 set_mon_vals no callback set 2024-03-29T10:47:35.104 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:35.116+0000 7f032250e640 10 monclient: shutdown 2024-03-29T10:47:35.104 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:35.116+0000 7f032250e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:47:35.104 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:35.116+0000 7f032250e640 1 -- 172.21.15.88:0/3109781761 >> v1:172.21.15.88:6789/0 conn(0x7f031c0a9ac0 legacy=0x7f031c0adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:35.104 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:35.117+0000 7f032250e640 1 -- 172.21.15.88:0/3109781761 shutdown_connections 2024-03-29T10:47:35.104 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:35.117+0000 7f032250e640 1 -- 172.21.15.88:0/3109781761 >> 172.21.15.88:0/3109781761 conn(0x7f031c0b35e0 msgr2=0x7f031c0b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:47:35.104 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:35.117+0000 7f032250e640 1 -- 172.21.15.88:0/3109781761 shutdown_connections 2024-03-29T10:47:35.104 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:35.117+0000 7f032250e640 1 -- 172.21.15.88:0/3109781761 wait complete. 2024-03-29T10:47:35.104 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:35.118+0000 7f032250e640 10 monclient: build_initial_monmap 2024-03-29T10:47:35.104 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:35.118+0000 7f032250e640 10 monclient: monmap: 2024-03-29T10:47:35.104 INFO:teuthology.misc.health.smithi088.stderr:epoch 0 2024-03-29T10:47:35.104 INFO:teuthology.misc.health.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:47:35.104 INFO:teuthology.misc.health.smithi088.stderr:last_changed 0.000000 2024-03-29T10:47:35.104 INFO:teuthology.misc.health.smithi088.stderr:created 0.000000 2024-03-29T10:47:35.104 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 0 (unknown) 2024-03-29T10:47:35.105 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:47:35.105 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-03-29T10:47:35.105 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-29T10:47:35.105 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-c 2024-03-29T10:47:35.105 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:47:35.105 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:35.118+0000 7f032250e640 1 librados: starting msgr at 2024-03-29T10:47:35.105 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:35.118+0000 7f032250e640 1 librados: starting objecter 2024-03-29T10:47:35.105 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:35.119+0000 7f032250e640 1 Processor -- start 2024-03-29T10:47:35.105 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:35.119+0000 7f032250e640 1 -- start start 2024-03-29T10:47:35.105 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:35.119+0000 7f032250e640 1 librados: setting wanted keys 2024-03-29T10:47:35.105 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:35.119+0000 7f032250e640 1 librados: calling monclient init 2024-03-29T10:47:35.105 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:35.119+0000 7f032250e640 10 monclient: init 2024-03-29T10:47:35.106 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:35.120+0000 7f032250e640 10 monclient: _reopen_session rank -1 2024-03-29T10:47:35.106 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:35.120+0000 7f032250e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:47:35.106 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:35.120+0000 7f032250e640 10 monclient(hunting): picked mon.noname-a con 0x7f031c0a9ac0 addr v1:172.21.15.88:6789/0 2024-03-29T10:47:35.106 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:35.121+0000 7f032250e640 10 monclient(hunting): picked mon.noname-b con 0x7f031c0af900 addr v1:172.21.15.196:6789/0 2024-03-29T10:47:35.107 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:35.124+0000 7f032250e640 10 monclient(hunting): picked mon.noname-c con 0x7f031c13ee90 addr v1:172.21.15.88:6790/0 2024-03-29T10:47:35.107 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:35.124+0000 7f032250e640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f031c0724f0 con 0x7f031c0a9ac0 2024-03-29T10:47:35.107 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:35.124+0000 7f032250e640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f031c070720 con 0x7f031c0af900 2024-03-29T10:47:35.107 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:35.124+0000 7f032250e640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f031c070a00 con 0x7f031c13ee90 2024-03-29T10:47:35.107 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:35.124+0000 7f032250e640 10 monclient(hunting): _renew_subs 2024-03-29T10:47:35.107 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:35.124+0000 7f032250e640 10 monclient(hunting): authenticate will time out at 1177.360116s 2024-03-29T10:47:35.107 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:35.124+0000 7f031bfff640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f031c0a9ac0 0x7f031c1dee00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:47667/0 (socket says 172.21.15.88:47667) 2024-03-29T10:47:35.107 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:35.124+0000 7f031bfff640 1 -- 172.21.15.88:0/1928801748 learned_addr learned my addr 172.21.15.88:0/1928801748 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-29T10:47:35.107 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:35.125+0000 7f0318ff9640 10 client.?.objecter ms_handle_connect 0x7f031c0a9ac0 2024-03-29T10:47:35.107 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:35.125+0000 7f0318ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:47:35.110 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:35.125+0000 7f0318ff9640 1 -- 172.21.15.88:0/1928801748 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1816766628 0 0) 0x7f031c0724f0 con 0x7f031c0a9ac0 2024-03-29T10:47:35.110 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:35.125+0000 7f0318ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:47:35.110 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:35.125+0000 7f0318ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:35.110 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:35.125+0000 7f0318ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:35.110 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:35.125+0000 7f0318ff9640 10 monclient(hunting): my global_id is 4476 2024-03-29T10:47:35.110 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:35.125+0000 7f0318ff9640 1 -- 172.21.15.88:0/1928801748 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0308004eb0 con 0x7f031c0a9ac0 2024-03-29T10:47:35.110 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:35.126+0000 7f0318ff9640 10 client.?.objecter ms_handle_connect 0x7f031c0af900 2024-03-29T10:47:35.110 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:35.126+0000 7f0318ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:47:35.110 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:35.126+0000 7f0318ff9640 1 -- 172.21.15.88:0/1928801748 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3521036043 0 0) 0x7f0308004eb0 con 0x7f031c0a9ac0 2024-03-29T10:47:35.110 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:35.126+0000 7f0318ff9640 1 -- 172.21.15.88:0/1928801748 --> v1:172.21.15.88:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0308005300 con 0x7f031c0a9ac0 2024-03-29T10:47:35.110 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:35.126+0000 7f0318ff9640 1 -- 172.21.15.88:0/1928801748 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f03140014b0 con 0x7f031c0a9ac0 2024-03-29T10:47:35.110 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:35.126+0000 7f0318ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:35.111 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:35.126+0000 7f0318ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:47:35.111 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:35.126+0000 7f0318ff9640 10 monclient(hunting): dump: 2024-03-29T10:47:35.111 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:47:35.111 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:47:35.111 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:47:35.111 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:47:35.111 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:35.111 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:47:35.111 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:47:35.111 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:47:35.111 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:47:35.111 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:47:35.111 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:35.126+0000 7f0318ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:47:35.111 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:35.126+0000 7f0318ff9640 1 -- 172.21.15.88:0/1928801748 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2941265731 0 0) 0x7f031c070720 con 0x7f031c0af900 2024-03-29T10:47:35.112 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:35.126+0000 7f0318ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:47:35.112 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:35.126+0000 7f0318ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:35.112 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:35.126+0000 7f0318ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:35.112 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:35.126+0000 7f0318ff9640 10 monclient(hunting): my global_id is 4465 2024-03-29T10:47:35.112 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:35.126+0000 7f0318ff9640 1 -- 172.21.15.88:0/1928801748 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0308005850 con 0x7f031c0af900 2024-03-29T10:47:35.112 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:35.129+0000 7f0318ff9640 10 client.?.objecter ms_handle_connect 0x7f031c13ee90 2024-03-29T10:47:35.112 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:35.129+0000 7f0318ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:47:35.112 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:35.129+0000 7f0318ff9640 1 -- 172.21.15.88:0/1928801748 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2283336194 0 0) 0x7f0308005850 con 0x7f031c0af900 2024-03-29T10:47:35.113 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:35.129+0000 7f0318ff9640 1 -- 172.21.15.88:0/1928801748 --> v1:172.21.15.196:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0308005ce0 con 0x7f031c0af900 2024-03-29T10:47:35.113 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:35.129+0000 7f0318ff9640 1 -- 172.21.15.88:0/1928801748 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f0310001680 con 0x7f031c0af900 2024-03-29T10:47:35.113 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:35.129+0000 7f0318ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:35.113 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:35.129+0000 7f0318ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T10:47:35.113 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:35.129+0000 7f0318ff9640 10 monclient(hunting): dump: 2024-03-29T10:47:35.113 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:47:35.113 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:47:35.113 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:47:35.113 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:47:35.113 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:35.113 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:47:35.113 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:47:35.113 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:47:35.113 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:47:35.114 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:47:35.114 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:35.130+0000 7f0318ff9640 1 -- 172.21.15.88:0/1928801748 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2281684075 0 0) 0x7f031c070a00 con 0x7f031c13ee90 2024-03-29T10:47:35.114 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:35.130+0000 7f0318ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:47:35.114 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:35.130+0000 7f0318ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:35.114 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:35.130+0000 7f0318ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:35.114 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:35.130+0000 7f0318ff9640 10 monclient(hunting): my global_id is 4511 2024-03-29T10:47:35.114 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:35.130+0000 7f0318ff9640 1 -- 172.21.15.88:0/1928801748 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0308005850 con 0x7f031c13ee90 2024-03-29T10:47:35.114 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:35.130+0000 7f0318ff9640 1 -- 172.21.15.88:0/1928801748 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3015407086 0 0) 0x7f0308005850 con 0x7f031c13ee90 2024-03-29T10:47:35.115 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:35.130+0000 7f0318ff9640 1 -- 172.21.15.88:0/1928801748 --> v1:172.21.15.88:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f031c070a00 con 0x7f031c13ee90 2024-03-29T10:47:35.115 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:35.130+0000 7f0318ff9640 1 -- 172.21.15.88:0/1928801748 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f030c001be0 con 0x7f031c13ee90 2024-03-29T10:47:35.115 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:35.130+0000 7f0318ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:35.115 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:35.130+0000 7f0318ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:47:35.115 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:35.130+0000 7f0318ff9640 10 monclient(hunting): dump: 2024-03-29T10:47:35.115 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:47:35.115 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:47:35.115 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:47:35.115 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:47:35.115 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:35.115 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:47:35.115 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:47:35.115 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:47:35.116 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:47:35.116 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:47:35.116 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:35.130+0000 7f0318ff9640 1 -- 172.21.15.88:0/1928801748 <== mon.1 v1:172.21.15.196:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4002039039 0 0) 0x7f0308005ce0 con 0x7f031c0af900 2024-03-29T10:47:35.116 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:35.130+0000 7f0318ff9640 1 -- 172.21.15.88:0/1928801748 >> v1:172.21.15.88:6790/0 conn(0x7f031c13ee90 legacy=0x7f031c1e60d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:35.116 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:35.130+0000 7f0318ff9640 1 -- 172.21.15.88:0/1928801748 >> v1:172.21.15.88:6789/0 conn(0x7f031c0a9ac0 legacy=0x7f031c1dee00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:35.116 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:35.130+0000 7f0318ff9640 10 monclient: _finish_hunting 0 2024-03-29T10:47:35.116 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:35.130+0000 7f0318ff9640 1 monclient: found mon.b 2024-03-29T10:47:35.116 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:35.130+0000 7f0318ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:47:35.116 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:35.130+0000 7f0318ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T10:47:35.116 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:35.130+0000 7f0318ff9640 1 -- 172.21.15.88:0/1928801748 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f031c072110 con 0x7f031c0af900 2024-03-29T10:47:35.116 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:35.131+0000 7f0318ff9640 10 monclient: _finish_auth 0 2024-03-29T10:47:35.116 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:35.131+0000 7f0318ff9640 10 monclient: _check_auth_tickets 2024-03-29T10:47:35.116 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:35.131+0000 7f0318ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:47:35.116 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:35.131+0000 7f032250e640 5 monclient: authenticate success, global_id 4465 2024-03-29T10:47:35.117 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:35.131+0000 7f032250e640 10 monclient: _renew_subs 2024-03-29T10:47:35.117 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:35.131+0000 7f032250e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T10:47:35.117 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:35.131+0000 7f032250e640 1 -- 172.21.15.88:0/1928801748 --> v1:172.21.15.196:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f031c06e8a0 con 0x7f031c0af900 2024-03-29T10:47:35.117 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:35.131+0000 7f032250e640 10 client.4465.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:47:35.117 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:35.131+0000 7f032250e640 10 monclient: _renew_subs 2024-03-29T10:47:35.117 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:35.131+0000 7f032250e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T10:47:35.117 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:35.131+0000 7f032250e640 1 -- 172.21.15.88:0/1928801748 --> v1:172.21.15.196:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f031c06f470 con 0x7f031c0af900 2024-03-29T10:47:35.117 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:35.131+0000 7f032250e640 1 librados: init done 2024-03-29T10:47:35.118 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:35.132+0000 7f032250e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:35.119 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:35.132+0000 7f032250e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:35.119 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:35.132+0000 7f032250e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T10:47:35.119 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:35.132+0000 7f032250e640 1 -- 172.21.15.88:0/1928801748 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f031c075f10 con 0x7f031c0af900 2024-03-29T10:47:35.119 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:35.133+0000 7f0318ff9640 1 -- 172.21.15.88:0/1928801748 <== mon.1 v1:172.21.15.196:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0310001b90 con 0x7f031c0af900 2024-03-29T10:47:35.119 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:35.133+0000 7f0318ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:47:35.119 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:35.133+0000 7f0318ff9640 1 -- 172.21.15.88:0/1928801748 <== mon.1 v1:172.21.15.196:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f0310003790 con 0x7f031c0af900 2024-03-29T10:47:35.119 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:35.133+0000 7f0318ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:35.119 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:35.133+0000 7f0318ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:47:35.119 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:35.133+0000 7f0318ff9640 10 monclient: dump: 2024-03-29T10:47:35.119 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:47:35.119 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:47:35.120 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:47:35.120 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:47:35.120 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:35.120 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:47:35.120 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:47:35.120 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:47:35.120 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:47:35.120 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:47:35.120 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:35.134+0000 7f031a7fc640 4 set_mon_vals no callback set 2024-03-29T10:47:35.120 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:35.135+0000 7f0318ff9640 1 -- 172.21.15.88:0/1928801748 <== mon.1 v1:172.21.15.196:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4235629220 0 0) 0x7f0310019760 con 0x7f031c0af900 2024-03-29T10:47:35.120 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:35.136+0000 7f0318ff9640 1 -- 172.21.15.88:0/1928801748 <== mon.1 v1:172.21.15.196:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4257673582 0 0) 0x7f03100834d0 con 0x7f031c0af900 2024-03-29T10:47:35.121 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:35.136+0000 7f0318ff9640 10 client.4465.objecter ms_dispatch 0x7f031c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:47:35.121 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:35.136+0000 7f0318ff9640 3 client.4465.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:47:35.121 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:35.136+0000 7f0318ff9640 3 client.4465.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:47:35.121 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:35.136+0000 7f0318ff9640 20 client.4465.objecter dump_active .. 0 homeless 2024-03-29T10:47:35.121 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:35.136+0000 7f031bfff640 10 monclient: get_auth_request con 0x7f030806e9e0 auth_method 0 2024-03-29T10:47:35.121 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:35.136+0000 7f0318ff9640 10 client.4465.objecter ms_handle_connect 0x7f030806e9e0 2024-03-29T10:47:35.124 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:35.141+0000 7f0318ff9640 1 -- 172.21.15.88:0/1928801748 <== mon.1 v1:172.21.15.196:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f031004edb0 con 0x7f031c0af900 2024-03-29T10:47:35.124 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:35.141+0000 7f0318ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:35.124 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:35.141+0000 7f0318ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:47:35.449 INFO:teuthology.misc.health.smithi088.stdout: 2024-03-29T10:47:35.449 INFO:teuthology.misc.health.smithi088.stdout:{"epoch":10,"fsid":"869f5188-f92b-42dd-a46d-8c646e0388a9","created":"2024-03-29T10:40:40.285087+0000","modified":"2024-03-29T10:40:42.536230+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:40:42.536230+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a26c0c1c-8de0-438c-a3f6-3df10ea61574","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5de24915-8dec-49b6-8b16-f3170197c591","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ac054af3-d589-4565-9644-e6af75975c53","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3ec4f045-c714-4a22-bd1a-e07330b45ab3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"cddb9867-8e30-4951-9b13-3d1e78b9c45a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dc6d9911-785d-432f-8b53-56a1ccab302f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"77c88437-b587-496e-8b4d-1824ea8b5305","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ce963d96-5a8e-44c3-99de-c0dbebe234ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:47:35.449 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:35.464+0000 7f032250e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:47:35.449 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:35.465+0000 7f032250e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:47:35.449 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:35.465+0000 7f032250e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T10:47:35.449 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:35.465+0000 7f032250e640 1 -- 172.21.15.88:0/1928801748 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f031c0722a0 con 0x7f031c0af900 2024-03-29T10:47:35.450 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:35.466+0000 7f0318ff9640 1 -- 172.21.15.88:0/1928801748 <== mon.1 v1:172.21.15.196:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 344506374) 0x7f0310050ce0 con 0x7f031c0af900 2024-03-29T10:47:35.450 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:35.466+0000 7f0318ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:47:35.450 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:35.466+0000 7f0318ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:47:35.453 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:35.471+0000 7f032250e640 10 librados: watch_flush enter 2024-03-29T10:47:35.453 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:35.471+0000 7f032250e640 10 librados: watch_flush exit 2024-03-29T10:47:35.454 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:35.471+0000 7f032250e640 20 client.4465.objecter shutdown clearing up homeless session... 2024-03-29T10:47:35.454 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:35.471+0000 7f032250e640 10 client.4465.objecter successfully canceled tick 2024-03-29T10:47:35.454 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:35.472+0000 7f032250e640 1 -- 172.21.15.88:0/1928801748 >> v1:172.21.15.88:6816/1752861349 conn(0x7f030806e9e0 legacy=0x7f0308070ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:35.454 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:35.472+0000 7f032250e640 10 monclient: shutdown 2024-03-29T10:47:35.455 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:35.472+0000 7f032250e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:47:35.455 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:35.472+0000 7f032250e640 1 -- 172.21.15.88:0/1928801748 >> v1:172.21.15.196:6789/0 conn(0x7f031c0af900 legacy=0x7f031c1df510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:35.455 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:35.472+0000 7f032250e640 1 -- 172.21.15.88:0/1928801748 shutdown_connections 2024-03-29T10:47:35.455 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:35.472+0000 7f032250e640 1 -- 172.21.15.88:0/1928801748 >> 172.21.15.88:0/1928801748 conn(0x7f031c0b35e0 msgr2=0x7f031c0adee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:47:35.455 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:35.472+0000 7f032250e640 1 -- 172.21.15.88:0/1928801748 shutdown_connections 2024-03-29T10:47:35.455 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:35.472+0000 7f032250e640 1 -- 172.21.15.88:0/1928801748 wait complete. 2024-03-29T10:47:35.455 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:35.472+0000 7f032250e640 1 librados: shutdown 2024-03-29T10:47:35.473 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:47:42.280 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:47:42.434 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:42.452+0000 7f8bda38c640 10 monclient: get_monmap_and_config 2024-03-29T10:47:42.435 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:42.452+0000 7f8bda38c640 10 monclient: build_initial_monmap 2024-03-29T10:47:42.435 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:42.452+0000 7f8bda38c640 10 monclient: monmap: 2024-03-29T10:47:42.435 INFO:teuthology.misc.health.smithi088.stderr:epoch 0 2024-03-29T10:47:42.435 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:47:42.435 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:47:42.453441+0000 2024-03-29T10:47:42.435 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:47:42.453441+0000 2024-03-29T10:47:42.435 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 0 (unknown) 2024-03-29T10:47:42.435 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:47:42.435 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-03-29T10:47:42.435 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-29T10:47:42.435 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-b 2024-03-29T10:47:42.436 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:47:42.437 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:42.455+0000 7f8bda38c640 1 Processor -- start 2024-03-29T10:47:42.440 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:42.457+0000 7f8bda38c640 1 -- start start 2024-03-29T10:47:42.440 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:42.457+0000 7f8bda38c640 10 monclient: init 2024-03-29T10:47:42.440 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:42.458+0000 7f8bda38c640 10 monclient: _reopen_session rank -1 2024-03-29T10:47:42.440 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:42.458+0000 7f8bda38c640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:47:42.440 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:42.458+0000 7f8bda38c640 10 monclient(hunting): picked mon.noname-b con 0x7f8bd41457f0 addr v1:172.21.15.88:6790/0 2024-03-29T10:47:42.441 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:42.458+0000 7f8bda38c640 10 monclient(hunting): picked mon.noname-a con 0x7f8bd413f650 addr v1:172.21.15.88:6789/0 2024-03-29T10:47:42.441 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:42.458+0000 7f8bda38c640 10 monclient(hunting): picked mon.noname-c con 0x7f8bd40a9d70 addr v1:172.21.15.196:6789/0 2024-03-29T10:47:42.441 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:42.458+0000 7f8bda38c640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8bd40aaef0 con 0x7f8bd413f650 2024-03-29T10:47:42.441 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:42.458+0000 7f8bda38c640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8bd4145bd0 con 0x7f8bd40a9d70 2024-03-29T10:47:42.441 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:42.458+0000 7f8bda38c640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8bd4145eb0 con 0x7f8bd41457f0 2024-03-29T10:47:42.441 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:42.458+0000 7f8bda38c640 10 monclient(hunting): _renew_subs 2024-03-29T10:47:42.441 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:42.458+0000 7f8bda38c640 10 monclient(hunting): authenticate will time out at 1184.694601s 2024-03-29T10:47:42.446 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:42.463+0000 7f8bd8929640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f8bd40a9d70 0x7f8bd414d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.88:51739/0 (socket says 172.21.15.88:51739) 2024-03-29T10:47:42.446 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:42.463+0000 7f8bd8929640 1 -- 172.21.15.88:0/530716978 learned_addr learned my addr 172.21.15.88:0/530716978 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-29T10:47:42.447 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:42.464+0000 7f8bd2ffd640 1 -- 172.21.15.88:0/530716978 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 698737258 0 0) 0x7f8bd4145bd0 con 0x7f8bd40a9d70 2024-03-29T10:47:42.447 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:42.464+0000 7f8bd2ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:47:42.447 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:42.464+0000 7f8bd2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:42.447 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:42.464+0000 7f8bd2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:42.447 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:42.464+0000 7f8bd2ffd640 10 monclient(hunting): my global_id is 4468 2024-03-29T10:47:42.447 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:42.464+0000 7f8bd2ffd640 1 -- 172.21.15.88:0/530716978 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8bbc003520 con 0x7f8bd40a9d70 2024-03-29T10:47:42.448 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:42.465+0000 7f8bd2ffd640 1 -- 172.21.15.88:0/530716978 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1628272000 0 0) 0x7f8bbc003520 con 0x7f8bd40a9d70 2024-03-29T10:47:42.448 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:42.465+0000 7f8bd2ffd640 1 -- 172.21.15.88:0/530716978 >> v1:172.21.15.88:6790/0 conn(0x7f8bd41457f0 legacy=0x7f8bd413ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:47:42.448 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:42.465+0000 7f8bd2ffd640 1 -- 172.21.15.88:0/530716978 >> v1:172.21.15.88:6789/0 conn(0x7f8bd413f650 legacy=0x7f8bd40b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:42.448 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:42.465+0000 7f8bd2ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:47:42.448 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:42.465+0000 7f8bd2ffd640 1 monclient: found mon.noname-c 2024-03-29T10:47:42.448 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:42.465+0000 7f8bd2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:47:42.448 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:42.465+0000 7f8bd2ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.196:6789/0 2024-03-29T10:47:42.448 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:42.465+0000 7f8bd2ffd640 1 -- 172.21.15.88:0/530716978 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bd4146340 con 0x7f8bd40a9d70 2024-03-29T10:47:42.448 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:42.465+0000 7f8bd2ffd640 10 monclient: _finish_auth 0 2024-03-29T10:47:42.448 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:42.465+0000 7f8bd2ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:47:42.448 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:42.465+0000 7f8bd2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:47:42.448 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:42.465+0000 7f8bda38c640 5 monclient: authenticate success, global_id 4468 2024-03-29T10:47:42.448 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:42.465+0000 7f8bda38c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:47:42.449 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:42.465+0000 7f8bd2ffd640 1 -- 172.21.15.88:0/530716978 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f8bc4001690 con 0x7f8bd40a9d70 2024-03-29T10:47:42.449 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:42.465+0000 7f8bd2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:42.449 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:42.465+0000 7f8bd2ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:47:42.449 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:42.465+0000 7f8bd2ffd640 10 monclient: dump: 2024-03-29T10:47:42.449 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:47:42.449 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:47:42.449 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:47:42.449 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:47:42.449 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:42.449 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:47:42.449 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:47:42.449 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:47:42.449 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:47:42.449 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:47:42.449 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:42.465+0000 7f8bda38c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:47:42.450 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:42.465+0000 7f8bd2ffd640 1 -- 172.21.15.88:0/530716978 <== mon.1 v1:172.21.15.196:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8bc4001860 con 0x7f8bd40a9d70 2024-03-29T10:47:42.450 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:42.465+0000 7f8bd2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:47:42.450 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:42.465+0000 7f8bd2ffd640 1 -- 172.21.15.88:0/530716978 <== mon.1 v1:172.21.15.196:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f8bc4001cb0 con 0x7f8bd40a9d70 2024-03-29T10:47:42.450 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:42.465+0000 7f8bd2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:42.450 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:42.465+0000 7f8bd2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:47:42.450 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:42.466+0000 7f8bd2ffd640 10 monclient: dump: 2024-03-29T10:47:42.450 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:47:42.450 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:47:42.450 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:47:42.450 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:47:42.450 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:42.450 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:47:42.451 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:47:42.451 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:47:42.451 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:47:42.451 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:47:42.452 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:42.466+0000 7f8bda38c640 10 monclient: get_monmap_and_config success 2024-03-29T10:47:42.452 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:42.466+0000 7f8bda38c640 4 set_mon_vals no callback set 2024-03-29T10:47:42.452 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:42.467+0000 7f8bda38c640 10 monclient: shutdown 2024-03-29T10:47:42.452 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:42.467+0000 7f8bda38c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:47:42.452 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:42.467+0000 7f8bda38c640 1 -- 172.21.15.88:0/530716978 >> v1:172.21.15.196:6789/0 conn(0x7f8bd40a9d70 legacy=0x7f8bd414d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:42.452 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:42.468+0000 7f8bda38c640 1 -- 172.21.15.88:0/530716978 shutdown_connections 2024-03-29T10:47:42.453 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:42.468+0000 7f8bda38c640 1 -- 172.21.15.88:0/530716978 >> 172.21.15.88:0/530716978 conn(0x7f8bd4066a00 msgr2=0x7f8bd4066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:47:42.453 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:42.470+0000 7f8bda38c640 1 -- 172.21.15.88:0/530716978 shutdown_connections 2024-03-29T10:47:42.455 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:42.470+0000 7f8bda38c640 1 -- 172.21.15.88:0/530716978 wait complete. 2024-03-29T10:47:42.455 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:42.471+0000 7f8bda38c640 10 monclient: build_initial_monmap 2024-03-29T10:47:42.455 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:42.472+0000 7f8bda38c640 10 monclient: monmap: 2024-03-29T10:47:42.455 INFO:teuthology.misc.health.smithi088.stderr:epoch 0 2024-03-29T10:47:42.456 INFO:teuthology.misc.health.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:47:42.456 INFO:teuthology.misc.health.smithi088.stderr:last_changed 0.000000 2024-03-29T10:47:42.456 INFO:teuthology.misc.health.smithi088.stderr:created 0.000000 2024-03-29T10:47:42.456 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 0 (unknown) 2024-03-29T10:47:42.456 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:47:42.456 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-03-29T10:47:42.456 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-29T10:47:42.456 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-c 2024-03-29T10:47:42.456 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:47:42.456 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:42.472+0000 7f8bda38c640 1 librados: starting msgr at 2024-03-29T10:47:42.456 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:42.472+0000 7f8bda38c640 1 librados: starting objecter 2024-03-29T10:47:42.456 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:42.473+0000 7f8bda38c640 1 Processor -- start 2024-03-29T10:47:42.457 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:42.473+0000 7f8bda38c640 1 -- start start 2024-03-29T10:47:42.458 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:42.473+0000 7f8bda38c640 1 librados: setting wanted keys 2024-03-29T10:47:42.458 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:42.473+0000 7f8bda38c640 1 librados: calling monclient init 2024-03-29T10:47:42.458 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:42.473+0000 7f8bda38c640 10 monclient: init 2024-03-29T10:47:42.458 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:42.474+0000 7f8bda38c640 10 monclient: _reopen_session rank -1 2024-03-29T10:47:42.458 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:42.474+0000 7f8bda38c640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:47:42.458 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:42.474+0000 7f8bda38c640 10 monclient(hunting): picked mon.noname-c con 0x7f8bd40a9d70 addr v1:172.21.15.88:6790/0 2024-03-29T10:47:42.458 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:42.474+0000 7f8bda38c640 10 monclient(hunting): picked mon.noname-a con 0x7f8bd413f650 addr v1:172.21.15.88:6789/0 2024-03-29T10:47:42.458 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:42.474+0000 7f8bda38c640 10 monclient(hunting): picked mon.noname-b con 0x7f8bd41457f0 addr v1:172.21.15.196:6789/0 2024-03-29T10:47:42.458 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:42.474+0000 7f8bda38c640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8bd4140160 con 0x7f8bd413f650 2024-03-29T10:47:42.458 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:42.474+0000 7f8bda38c640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8bd41cccb0 con 0x7f8bd41457f0 2024-03-29T10:47:42.458 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:42.474+0000 7f8bda38c640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8bd41ccf20 con 0x7f8bd40a9d70 2024-03-29T10:47:42.458 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:42.475+0000 7f8bda38c640 10 monclient(hunting): _renew_subs 2024-03-29T10:47:42.459 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:42.475+0000 7f8bda38c640 10 monclient(hunting): authenticate will time out at 1184.710959s 2024-03-29T10:47:42.462 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:42.479+0000 7f8bd3fff640 1 --1- >> v1:172.21.15.88:6790/0 conn(0x7f8bd40a9d70 0x7f8bd41405f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6790/0 says I am v1:172.21.15.88:42359/0 (socket says 172.21.15.88:42359) 2024-03-29T10:47:42.463 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:42.479+0000 7f8bd3fff640 1 -- 172.21.15.88:0/1288514118 learned_addr learned my addr 172.21.15.88:0/1288514118 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-29T10:47:42.469 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:42.482+0000 7f8bd0ff9640 10 client.?.objecter ms_handle_connect 0x7f8bd40a9d70 2024-03-29T10:47:42.469 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:42.482+0000 7f8bd0ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:47:42.469 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:42.482+0000 7f8bd0ff9640 10 client.?.objecter ms_handle_connect 0x7f8bd413f650 2024-03-29T10:47:42.469 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:42.482+0000 7f8bd0ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:47:42.469 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:42.482+0000 7f8bd0ff9640 10 client.?.objecter ms_handle_connect 0x7f8bd41457f0 2024-03-29T10:47:42.469 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:42.482+0000 7f8bd0ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:47:42.469 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:42.482+0000 7f8bd0ff9640 1 -- 172.21.15.88:0/1288514118 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1188606064 0 0) 0x7f8bd4140160 con 0x7f8bd413f650 2024-03-29T10:47:42.470 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:42.482+0000 7f8bd0ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:47:42.470 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:42.482+0000 7f8bd0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:42.470 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:42.482+0000 7f8bd0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:42.470 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:42.482+0000 7f8bd0ff9640 10 monclient(hunting): my global_id is 4482 2024-03-29T10:47:42.470 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:42.482+0000 7f8bd0ff9640 1 -- 172.21.15.88:0/1288514118 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8bc0004eb0 con 0x7f8bd413f650 2024-03-29T10:47:42.470 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:42.482+0000 7f8bd0ff9640 1 -- 172.21.15.88:0/1288514118 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3512375763 0 0) 0x7f8bd41ccf20 con 0x7f8bd40a9d70 2024-03-29T10:47:42.470 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:42.482+0000 7f8bd0ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:47:42.470 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:42.482+0000 7f8bd0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:42.470 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:42.482+0000 7f8bd0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:42.471 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:42.482+0000 7f8bd0ff9640 10 monclient(hunting): my global_id is 4514 2024-03-29T10:47:42.471 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:42.482+0000 7f8bd0ff9640 1 -- 172.21.15.88:0/1288514118 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8bd4140160 con 0x7f8bd40a9d70 2024-03-29T10:47:42.471 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:42.482+0000 7f8bd0ff9640 1 -- 172.21.15.88:0/1288514118 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1424878678 0 0) 0x7f8bd41cccb0 con 0x7f8bd41457f0 2024-03-29T10:47:42.471 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:42.482+0000 7f8bd0ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:47:42.471 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:42.482+0000 7f8bd0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:42.471 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:42.482+0000 7f8bd0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:42.471 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:42.482+0000 7f8bd0ff9640 10 monclient(hunting): my global_id is 4471 2024-03-29T10:47:42.471 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:42.482+0000 7f8bd0ff9640 1 -- 172.21.15.88:0/1288514118 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8bd41ccf20 con 0x7f8bd41457f0 2024-03-29T10:47:42.471 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:42.483+0000 7f8bd0ff9640 1 -- 172.21.15.88:0/1288514118 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4042811350 0 0) 0x7f8bc0004eb0 con 0x7f8bd413f650 2024-03-29T10:47:42.471 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:42.483+0000 7f8bd0ff9640 1 -- 172.21.15.88:0/1288514118 --> v1:172.21.15.88:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8bd41cccb0 con 0x7f8bd413f650 2024-03-29T10:47:42.471 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:42.483+0000 7f8bd0ff9640 1 -- 172.21.15.88:0/1288514118 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f8bc80018b0 con 0x7f8bd413f650 2024-03-29T10:47:42.472 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:42.483+0000 7f8bd0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:42.472 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:42.483+0000 7f8bd0ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:47:42.472 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:42.483+0000 7f8bd0ff9640 10 monclient(hunting): dump: 2024-03-29T10:47:42.472 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:47:42.472 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:47:42.472 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:47:42.472 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:47:42.472 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:42.472 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:47:42.472 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:47:42.473 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:47:42.473 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:47:42.473 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:47:42.473 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:42.483+0000 7f8bd0ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:47:42.473 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:42.483+0000 7f8bd0ff9640 1 -- 172.21.15.88:0/1288514118 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2980247189 0 0) 0x7f8bd4140160 con 0x7f8bd40a9d70 2024-03-29T10:47:42.473 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:42.483+0000 7f8bd0ff9640 1 -- 172.21.15.88:0/1288514118 --> v1:172.21.15.88:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8bc0004eb0 con 0x7f8bd40a9d70 2024-03-29T10:47:42.473 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:42.483+0000 7f8bd0ff9640 1 -- 172.21.15.88:0/1288514118 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f8bcc004260 con 0x7f8bd40a9d70 2024-03-29T10:47:42.473 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:42.483+0000 7f8bd0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:42.473 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:42.483+0000 7f8bd0ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:47:42.473 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:42.483+0000 7f8bd0ff9640 10 monclient(hunting): dump: 2024-03-29T10:47:42.473 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:47:42.474 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:47:42.474 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:47:42.474 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:47:42.474 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:42.474 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:47:42.474 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:47:42.474 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:47:42.474 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:47:42.474 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:47:42.474 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:42.483+0000 7f8bd0ff9640 1 -- 172.21.15.88:0/1288514118 <== mon.0 v1:172.21.15.88:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1524942792 0 0) 0x7f8bd41cccb0 con 0x7f8bd413f650 2024-03-29T10:47:42.474 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:42.483+0000 7f8bd0ff9640 1 -- 172.21.15.88:0/1288514118 >> v1:172.21.15.88:6790/0 conn(0x7f8bd40a9d70 legacy=0x7f8bd41405f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:42.474 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:42.483+0000 7f8bd0ff9640 1 -- 172.21.15.88:0/1288514118 >> v1:172.21.15.196:6789/0 conn(0x7f8bd41457f0 legacy=0x7f8bd41e2050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:42.474 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:42.483+0000 7f8bd0ff9640 10 monclient: _finish_hunting 0 2024-03-29T10:47:42.474 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:42.483+0000 7f8bd0ff9640 1 monclient: found mon.a 2024-03-29T10:47:42.475 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:42.483+0000 7f8bd0ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:47:42.475 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:42.483+0000 7f8bd0ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-03-29T10:47:42.475 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:42.483+0000 7f8bd0ff9640 1 -- 172.21.15.88:0/1288514118 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bd413fda0 con 0x7f8bd413f650 2024-03-29T10:47:42.475 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:42.483+0000 7f8bd0ff9640 10 monclient: _finish_auth 0 2024-03-29T10:47:42.475 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:42.483+0000 7f8bd0ff9640 10 monclient: _check_auth_tickets 2024-03-29T10:47:42.475 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:42.483+0000 7f8bd0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:47:42.475 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:42.484+0000 7f8bda38c640 5 monclient: authenticate success, global_id 4482 2024-03-29T10:47:42.475 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:42.484+0000 7f8bda38c640 10 monclient: _renew_subs 2024-03-29T10:47:42.475 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:42.484+0000 7f8bda38c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-03-29T10:47:42.475 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:42.484+0000 7f8bda38c640 1 -- 172.21.15.88:0/1288514118 --> v1:172.21.15.88:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8bd41cb340 con 0x7f8bd413f650 2024-03-29T10:47:42.475 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:42.484+0000 7f8bda38c640 10 client.4482.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:47:42.475 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:42.484+0000 7f8bda38c640 10 monclient: _renew_subs 2024-03-29T10:47:42.475 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:42.484+0000 7f8bda38c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-03-29T10:47:42.475 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:42.484+0000 7f8bda38c640 1 -- 172.21.15.88:0/1288514118 --> v1:172.21.15.88:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8bd41cb6e0 con 0x7f8bd413f650 2024-03-29T10:47:42.475 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:42.484+0000 7f8bda38c640 1 librados: init done 2024-03-29T10:47:42.476 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:42.485+0000 7f8bd0ff9640 1 -- 172.21.15.88:0/1288514118 <== mon.0 v1:172.21.15.88:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8bc80035f0 con 0x7f8bd413f650 2024-03-29T10:47:42.476 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:42.485+0000 7f8bd0ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:47:42.476 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:42.485+0000 7f8bd0ff9640 1 -- 172.21.15.88:0/1288514118 <== mon.0 v1:172.21.15.88:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f8bc8003980 con 0x7f8bd413f650 2024-03-29T10:47:42.476 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:42.485+0000 7f8bd1ffb640 4 set_mon_vals no callback set 2024-03-29T10:47:42.476 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:42.486+0000 7f8bd0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:42.476 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:42.486+0000 7f8bd0ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:47:42.476 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:42.486+0000 7f8bd0ff9640 10 monclient: dump: 2024-03-29T10:47:42.476 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:47:42.476 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:47:42.476 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:47:42.476 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:47:42.476 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:42.476 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:47:42.476 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:47:42.477 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:47:42.477 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:47:42.477 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:47:42.478 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:42.487+0000 7f8bd0ff9640 1 -- 172.21.15.88:0/1288514118 <== mon.0 v1:172.21.15.88:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4235629220 0 0) 0x7f8bc8003ae0 con 0x7f8bd413f650 2024-03-29T10:47:42.478 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:42.488+0000 7f8bd0ff9640 1 -- 172.21.15.88:0/1288514118 <== mon.0 v1:172.21.15.88:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4257673582 0 0) 0x7f8bc8083810 con 0x7f8bd413f650 2024-03-29T10:47:42.478 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:42.488+0000 7f8bd0ff9640 10 client.4482.objecter ms_dispatch 0x7f8bd4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:47:42.478 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:42.488+0000 7f8bd0ff9640 3 client.4482.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:47:42.478 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:42.488+0000 7f8bd0ff9640 3 client.4482.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:47:42.478 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:42.488+0000 7f8bd3fff640 10 monclient: get_auth_request con 0x7f8bc006e120 auth_method 0 2024-03-29T10:47:42.478 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:42.488+0000 7f8bd0ff9640 20 client.4482.objecter dump_active .. 0 homeless 2024-03-29T10:47:42.478 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:42.489+0000 7f8bd0ff9640 10 client.4482.objecter ms_handle_connect 0x7f8bc006e120 2024-03-29T10:47:42.479 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:42.489+0000 7f8bda38c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:42.479 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:42.489+0000 7f8bda38c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:42.479 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:42.489+0000 7f8bda38c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-03-29T10:47:42.479 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:42.489+0000 7f8bda38c640 1 -- 172.21.15.88:0/1288514118 --> v1:172.21.15.88:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8bb4005080 con 0x7f8bd413f650 2024-03-29T10:47:42.480 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:42.497+0000 7f8bd0ff9640 1 -- 172.21.15.88:0/1288514118 <== mon.0 v1:172.21.15.88:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8bc804f0f0 con 0x7f8bd413f650 2024-03-29T10:47:42.480 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:42.497+0000 7f8bd0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:42.480 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:42.497+0000 7f8bd0ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:47:42.805 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:42.821+0000 7f8bda38c640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:47:42.805 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:42.822+0000 7f8bda38c640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:47:42.806 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:42.822+0000 7f8bda38c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-03-29T10:47:42.806 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:42.822+0000 7f8bda38c640 1 -- 172.21.15.88:0/1288514118 --> v1:172.21.15.88:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8bb4005330 con 0x7f8bd413f650 2024-03-29T10:47:42.808 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:42.823+0000 7f8bd0ff9640 1 -- 172.21.15.88:0/1288514118 <== mon.0 v1:172.21.15.88:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 344506374) 0x7f8bc8051020 con 0x7f8bd413f650 2024-03-29T10:47:42.808 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:42.823+0000 7f8bd0ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:47:42.808 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:42.823+0000 7f8bd0ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:47:42.808 INFO:teuthology.misc.health.smithi088.stdout: 2024-03-29T10:47:42.808 INFO:teuthology.misc.health.smithi088.stdout:{"epoch":10,"fsid":"869f5188-f92b-42dd-a46d-8c646e0388a9","created":"2024-03-29T10:40:40.285087+0000","modified":"2024-03-29T10:40:42.536230+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:40:42.536230+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a26c0c1c-8de0-438c-a3f6-3df10ea61574","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5de24915-8dec-49b6-8b16-f3170197c591","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ac054af3-d589-4565-9644-e6af75975c53","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3ec4f045-c714-4a22-bd1a-e07330b45ab3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"cddb9867-8e30-4951-9b13-3d1e78b9c45a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dc6d9911-785d-432f-8b53-56a1ccab302f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"77c88437-b587-496e-8b4d-1824ea8b5305","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ce963d96-5a8e-44c3-99de-c0dbebe234ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:47:42.809 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:42.827+0000 7f8bda38c640 10 librados: watch_flush enter 2024-03-29T10:47:42.809 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:42.827+0000 7f8bda38c640 10 librados: watch_flush exit 2024-03-29T10:47:42.809 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:42.827+0000 7f8bda38c640 20 client.4482.objecter shutdown clearing up homeless session... 2024-03-29T10:47:42.810 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:42.827+0000 7f8bda38c640 10 client.4482.objecter successfully canceled tick 2024-03-29T10:47:42.812 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:42.829+0000 7f8bda38c640 1 -- 172.21.15.88:0/1288514118 >> v1:172.21.15.88:6816/1752861349 conn(0x7f8bc006e120 legacy=0x7f8bc00705e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:42.812 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:42.829+0000 7f8bda38c640 10 monclient: shutdown 2024-03-29T10:47:42.812 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:42.829+0000 7f8bda38c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:47:42.813 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:42.829+0000 7f8bda38c640 1 -- 172.21.15.88:0/1288514118 >> v1:172.21.15.88:6789/0 conn(0x7f8bd413f650 legacy=0x7f8bd4140d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:42.813 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:42.830+0000 7f8bda38c640 1 -- 172.21.15.88:0/1288514118 shutdown_connections 2024-03-29T10:47:42.813 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:42.830+0000 7f8bda38c640 1 -- 172.21.15.88:0/1288514118 >> 172.21.15.88:0/1288514118 conn(0x7f8bd4066a00 msgr2=0x7f8bd4066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:47:42.813 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:42.831+0000 7f8bda38c640 1 -- 172.21.15.88:0/1288514118 shutdown_connections 2024-03-29T10:47:42.814 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:42.832+0000 7f8bda38c640 1 -- 172.21.15.88:0/1288514118 wait complete. 2024-03-29T10:47:42.814 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:42.832+0000 7f8bda38c640 1 librados: shutdown 2024-03-29T10:47:42.835 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:47:49.642 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:47:49.791 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:49.808+0000 7fcf27d23640 10 monclient: get_monmap_and_config 2024-03-29T10:47:49.791 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:49.808+0000 7fcf27d23640 10 monclient: build_initial_monmap 2024-03-29T10:47:49.791 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:49.808+0000 7fcf27d23640 10 monclient: monmap: 2024-03-29T10:47:49.791 INFO:teuthology.misc.health.smithi088.stderr:epoch 0 2024-03-29T10:47:49.791 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:47:49.792 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:47:49.809960+0000 2024-03-29T10:47:49.792 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:47:49.809960+0000 2024-03-29T10:47:49.792 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 0 (unknown) 2024-03-29T10:47:49.792 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:47:49.792 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-03-29T10:47:49.792 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-29T10:47:49.792 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-b 2024-03-29T10:47:49.792 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:47:49.793 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:49.810+0000 7fcf27d23640 1 Processor -- start 2024-03-29T10:47:49.793 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:49.810+0000 7fcf27d23640 1 -- start start 2024-03-29T10:47:49.793 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:49.810+0000 7fcf27d23640 10 monclient: init 2024-03-29T10:47:49.795 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:49.811+0000 7fcf27d23640 10 monclient: _reopen_session rank -1 2024-03-29T10:47:49.796 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:49.811+0000 7fcf27d23640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:47:49.796 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:49.811+0000 7fcf27d23640 10 monclient(hunting): picked mon.noname-b con 0x7fcf200b1f60 addr v1:172.21.15.88:6790/0 2024-03-29T10:47:49.796 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:49.811+0000 7fcf27d23640 10 monclient(hunting): picked mon.noname-a con 0x7fcf20071eb0 addr v1:172.21.15.88:6789/0 2024-03-29T10:47:49.796 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:49.811+0000 7fcf27d23640 10 monclient(hunting): picked mon.noname-c con 0x7fcf20143d10 addr v1:172.21.15.196:6789/0 2024-03-29T10:47:49.796 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:49.811+0000 7fcf27d23640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf2006ef80 con 0x7fcf20071eb0 2024-03-29T10:47:49.796 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:49.811+0000 7fcf27d23640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf2006f260 con 0x7fcf20143d10 2024-03-29T10:47:49.796 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:49.811+0000 7fcf27d23640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf200b40c0 con 0x7fcf200b1f60 2024-03-29T10:47:49.796 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:49.811+0000 7fcf27d23640 10 monclient(hunting): _renew_subs 2024-03-29T10:47:49.796 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:49.811+0000 7fcf27d23640 10 monclient(hunting): authenticate will time out at 1192.047297s 2024-03-29T10:47:49.797 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:49.815+0000 7fcf25abf640 1 --1- >> v1:172.21.15.88:6790/0 conn(0x7fcf200b1f60 0x7fcf20143600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6790/0 says I am v1:172.21.15.88:47969/0 (socket says 172.21.15.88:47969) 2024-03-29T10:47:49.797 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:49.815+0000 7fcf25abf640 1 -- 172.21.15.88:0/349532945 learned_addr learned my addr 172.21.15.88:0/349532945 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-29T10:47:49.799 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:49.817+0000 7fcf24abd640 1 -- 172.21.15.88:0/349532945 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 857979996 0 0) 0x7fcf200b40c0 con 0x7fcf200b1f60 2024-03-29T10:47:49.800 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:49.817+0000 7fcf24abd640 10 monclient(hunting): _negotiate 2024-03-29T10:47:49.800 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:49.817+0000 7fcf24abd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:49.800 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:49.817+0000 7fcf24abd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:49.800 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:49.817+0000 7fcf24abd640 10 monclient(hunting): my global_id is 4517 2024-03-29T10:47:49.800 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:49.817+0000 7fcf24abd640 1 -- 172.21.15.88:0/349532945 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcf0c003520 con 0x7fcf200b1f60 2024-03-29T10:47:49.800 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:49.818+0000 7fcf24abd640 1 -- 172.21.15.88:0/349532945 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 104496009 0 0) 0x7fcf2006ef80 con 0x7fcf20071eb0 2024-03-29T10:47:49.800 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:49.818+0000 7fcf24abd640 10 monclient(hunting): _negotiate 2024-03-29T10:47:49.803 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:49.818+0000 7fcf24abd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:49.803 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:49.818+0000 7fcf24abd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:49.803 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:49.818+0000 7fcf24abd640 10 monclient(hunting): my global_id is 4485 2024-03-29T10:47:49.803 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:49.818+0000 7fcf24abd640 1 -- 172.21.15.88:0/349532945 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcf200b40c0 con 0x7fcf20071eb0 2024-03-29T10:47:49.804 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:49.820+0000 7fcf24abd640 1 -- 172.21.15.88:0/349532945 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1744580820 0 0) 0x7fcf200b40c0 con 0x7fcf20071eb0 2024-03-29T10:47:49.804 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:49.820+0000 7fcf24abd640 1 -- 172.21.15.88:0/349532945 >> v1:172.21.15.88:6790/0 conn(0x7fcf200b1f60 legacy=0x7fcf20143600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:49.804 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:49.820+0000 7fcf24abd640 1 -- 172.21.15.88:0/349532945 >> v1:172.21.15.196:6789/0 conn(0x7fcf20143d10 legacy=0x7fcf20146b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:47:49.804 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:49.821+0000 7fcf24abd640 10 monclient: _finish_hunting 0 2024-03-29T10:47:49.804 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:49.821+0000 7fcf24abd640 1 monclient: found mon.noname-a 2024-03-29T10:47:49.804 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:49.821+0000 7fcf24abd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:47:49.804 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:49.821+0000 7fcf24abd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.88:6789/0 2024-03-29T10:47:49.804 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:49.821+0000 7fcf24abd640 1 -- 172.21.15.88:0/349532945 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf20155d70 con 0x7fcf20071eb0 2024-03-29T10:47:49.804 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:49.821+0000 7fcf24abd640 10 monclient: _finish_auth 0 2024-03-29T10:47:49.804 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:49.821+0000 7fcf24abd640 10 monclient: _check_auth_tickets 2024-03-29T10:47:49.804 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:49.821+0000 7fcf24abd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:47:49.804 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:49.821+0000 7fcf27d23640 5 monclient: authenticate success, global_id 4485 2024-03-29T10:47:49.804 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:49.821+0000 7fcf27d23640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:47:49.804 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:49.821+0000 7fcf24abd640 1 -- 172.21.15.88:0/349532945 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7fcf10001670 con 0x7fcf20071eb0 2024-03-29T10:47:49.805 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:49.821+0000 7fcf24abd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:49.805 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:49.821+0000 7fcf24abd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:47:49.805 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:49.821+0000 7fcf24abd640 10 monclient: dump: 2024-03-29T10:47:49.805 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:47:49.805 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:47:49.805 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:47:49.805 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:47:49.805 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:49.805 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:47:49.805 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:47:49.805 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:47:49.805 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:47:49.805 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:47:49.805 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:49.821+0000 7fcf27d23640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:47:49.805 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:49.821+0000 7fcf24abd640 1 -- 172.21.15.88:0/349532945 <== mon.0 v1:172.21.15.88:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcf100017d0 con 0x7fcf20071eb0 2024-03-29T10:47:49.806 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:49.821+0000 7fcf24abd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:47:49.806 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:49.821+0000 7fcf27d23640 10 monclient: get_monmap_and_config success 2024-03-29T10:47:49.806 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:49.821+0000 7fcf27d23640 4 set_mon_vals no callback set 2024-03-29T10:47:49.806 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:49.821+0000 7fcf24abd640 1 -- 172.21.15.88:0/349532945 <== mon.0 v1:172.21.15.88:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7fcf10001be0 con 0x7fcf20071eb0 2024-03-29T10:47:49.806 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:49.821+0000 7fcf24abd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:49.806 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:49.821+0000 7fcf24abd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:47:49.806 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:49.821+0000 7fcf24abd640 10 monclient: dump: 2024-03-29T10:47:49.806 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:47:49.806 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:47:49.806 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:47:49.806 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:47:49.806 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:49.806 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:47:49.806 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:47:49.807 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:47:49.807 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:47:49.807 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:47:49.807 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:49.822+0000 7fcf27d23640 10 monclient: shutdown 2024-03-29T10:47:49.807 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:49.822+0000 7fcf27d23640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:47:49.807 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:49.822+0000 7fcf27d23640 1 -- 172.21.15.88:0/349532945 >> v1:172.21.15.88:6789/0 conn(0x7fcf20071eb0 legacy=0x7fcf20146470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:49.808 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:49.824+0000 7fcf27d23640 1 -- 172.21.15.88:0/349532945 shutdown_connections 2024-03-29T10:47:49.808 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:49.824+0000 7fcf27d23640 1 -- 172.21.15.88:0/349532945 >> 172.21.15.88:0/349532945 conn(0x7fcf20066a00 msgr2=0x7fcf20066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:47:49.810 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:49.828+0000 7fcf27d23640 1 -- 172.21.15.88:0/349532945 shutdown_connections 2024-03-29T10:47:49.811 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:49.828+0000 7fcf27d23640 1 -- 172.21.15.88:0/349532945 wait complete. 2024-03-29T10:47:49.811 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:49.829+0000 7fcf27d23640 10 monclient: build_initial_monmap 2024-03-29T10:47:49.811 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:49.829+0000 7fcf27d23640 10 monclient: monmap: 2024-03-29T10:47:49.811 INFO:teuthology.misc.health.smithi088.stderr:epoch 0 2024-03-29T10:47:49.811 INFO:teuthology.misc.health.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:47:49.811 INFO:teuthology.misc.health.smithi088.stderr:last_changed 0.000000 2024-03-29T10:47:49.811 INFO:teuthology.misc.health.smithi088.stderr:created 0.000000 2024-03-29T10:47:49.811 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 0 (unknown) 2024-03-29T10:47:49.811 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:47:49.811 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-03-29T10:47:49.812 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-29T10:47:49.812 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-c 2024-03-29T10:47:49.812 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:47:49.812 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:49.829+0000 7fcf27d23640 1 librados: starting msgr at 2024-03-29T10:47:49.812 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:49.829+0000 7fcf27d23640 1 librados: starting objecter 2024-03-29T10:47:49.812 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:49.830+0000 7fcf27d23640 1 Processor -- start 2024-03-29T10:47:49.813 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:49.830+0000 7fcf27d23640 1 -- start start 2024-03-29T10:47:49.813 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:49.830+0000 7fcf27d23640 1 librados: setting wanted keys 2024-03-29T10:47:49.813 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:49.830+0000 7fcf27d23640 1 librados: calling monclient init 2024-03-29T10:47:49.813 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:49.830+0000 7fcf27d23640 10 monclient: init 2024-03-29T10:47:49.814 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:49.830+0000 7fcf27d23640 10 monclient: _reopen_session rank -1 2024-03-29T10:47:49.814 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:49.830+0000 7fcf27d23640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:47:49.827 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:49.833+0000 7fcf27d23640 10 monclient(hunting): picked mon.noname-a con 0x7fcf200b1f60 addr v1:172.21.15.88:6789/0 2024-03-29T10:47:49.827 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:49.833+0000 7fcf27d23640 10 monclient(hunting): picked mon.noname-b con 0x7fcf20143d10 addr v1:172.21.15.196:6789/0 2024-03-29T10:47:49.828 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:49.836+0000 7fcf25abf640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fcf200b1f60 0x7fcf20131920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:37975/0 (socket says 172.21.15.88:37975) 2024-03-29T10:47:49.828 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:49.837+0000 7fcf262c0640 1 --1- >> v1:172.21.15.88:6790/0 conn(0x7fcf20141400 0x7fcf20132030 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6790/0 says I am v1:172.21.15.88:48105/0 (socket says 172.21.15.88:48105) 2024-03-29T10:47:49.828 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:49.837+0000 7fcf27d23640 10 monclient(hunting): picked mon.noname-c con 0x7fcf20141400 addr v1:172.21.15.88:6790/0 2024-03-29T10:47:49.828 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:49.837+0000 7fcf25abf640 1 -- 172.21.15.88:0/4179066152 learned_addr learned my addr 172.21.15.88:0/4179066152 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-29T10:47:49.828 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:49.837+0000 7fcf167fc640 10 client.?.objecter ms_handle_connect 0x7fcf200b1f60 2024-03-29T10:47:49.828 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:49.837+0000 7fcf167fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:47:49.828 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:49.837+0000 7fcf167fc640 10 client.?.objecter ms_handle_connect 0x7fcf20141400 2024-03-29T10:47:49.828 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:49.837+0000 7fcf167fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:47:49.828 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:49.837+0000 7fcf27d23640 1 -- 172.21.15.88:0/4179066152 --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf201430d0 con 0x7fcf200b1f60 2024-03-29T10:47:49.828 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:49.837+0000 7fcf27d23640 1 -- 172.21.15.88:0/4179066152 --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf20132740 con 0x7fcf20143d10 2024-03-29T10:47:49.829 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:49.837+0000 7fcf27d23640 1 -- 172.21.15.88:0/4179066152 --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf20142110 con 0x7fcf20141400 2024-03-29T10:47:49.829 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:49.837+0000 7fcf27d23640 10 monclient(hunting): _renew_subs 2024-03-29T10:47:49.829 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:49.837+0000 7fcf27d23640 10 monclient(hunting): authenticate will time out at 1192.073835s 2024-03-29T10:47:49.829 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:49.838+0000 7fcf167fc640 10 client.?.objecter ms_handle_connect 0x7fcf20143d10 2024-03-29T10:47:49.829 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:49.838+0000 7fcf167fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:47:49.829 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:49.838+0000 7fcf167fc640 1 -- 172.21.15.88:0/4179066152 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3617788547 0 0) 0x7fcf20132740 con 0x7fcf20143d10 2024-03-29T10:47:49.829 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:49.838+0000 7fcf167fc640 10 monclient(hunting): _negotiate 2024-03-29T10:47:49.829 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:49.838+0000 7fcf167fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:49.829 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:49.838+0000 7fcf167fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:49.829 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:49.838+0000 7fcf167fc640 10 monclient(hunting): my global_id is 4474 2024-03-29T10:47:49.829 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:49.838+0000 7fcf167fc640 1 -- 172.21.15.88:0/4179066152 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcf04004eb0 con 0x7fcf20143d10 2024-03-29T10:47:49.829 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:49.839+0000 7fcf167fc640 1 -- 172.21.15.88:0/4179066152 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2710928278 0 0) 0x7fcf04004eb0 con 0x7fcf20143d10 2024-03-29T10:47:49.829 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:49.839+0000 7fcf167fc640 1 -- 172.21.15.88:0/4179066152 --> v1:172.21.15.196:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcf20132740 con 0x7fcf20143d10 2024-03-29T10:47:49.829 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:49.839+0000 7fcf167fc640 1 -- 172.21.15.88:0/4179066152 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7fcf10001ba0 con 0x7fcf20143d10 2024-03-29T10:47:49.830 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:49.839+0000 7fcf167fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:49.830 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:49.839+0000 7fcf167fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:47:49.830 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:49.839+0000 7fcf167fc640 10 monclient(hunting): dump: 2024-03-29T10:47:49.830 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:47:49.830 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:47:49.830 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:47:49.830 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:47:49.830 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:49.830 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:47:49.830 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:47:49.830 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:47:49.830 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:47:49.830 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:47:49.831 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:49.839+0000 7fcf167fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:47:49.831 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:49.839+0000 7fcf167fc640 1 -- 172.21.15.88:0/4179066152 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 803050447 0 0) 0x7fcf201430d0 con 0x7fcf200b1f60 2024-03-29T10:47:49.831 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:49.839+0000 7fcf167fc640 10 monclient(hunting): _negotiate 2024-03-29T10:47:49.831 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:49.839+0000 7fcf167fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:49.831 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:49.839+0000 7fcf167fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:49.831 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:49.839+0000 7fcf167fc640 10 monclient(hunting): my global_id is 4488 2024-03-29T10:47:49.831 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:49.839+0000 7fcf167fc640 1 -- 172.21.15.88:0/4179066152 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcf04004eb0 con 0x7fcf200b1f60 2024-03-29T10:47:49.831 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:49.840+0000 7fcf167fc640 1 -- 172.21.15.88:0/4179066152 <== mon.1 v1:172.21.15.196:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 239485018 0 0) 0x7fcf20132740 con 0x7fcf20143d10 2024-03-29T10:47:49.831 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:49.840+0000 7fcf167fc640 1 -- 172.21.15.88:0/4179066152 >> v1:172.21.15.88:6790/0 conn(0x7fcf20141400 legacy=0x7fcf20132030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:49.831 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:49.840+0000 7fcf167fc640 1 -- 172.21.15.88:0/4179066152 >> v1:172.21.15.88:6789/0 conn(0x7fcf200b1f60 legacy=0x7fcf20131920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:49.831 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:49.840+0000 7fcf167fc640 10 monclient: _finish_hunting 0 2024-03-29T10:47:49.831 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:49.840+0000 7fcf167fc640 1 monclient: found mon.b 2024-03-29T10:47:49.831 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:49.840+0000 7fcf167fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:47:49.831 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:49.840+0000 7fcf167fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T10:47:49.831 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:49.841+0000 7fcf167fc640 1 -- 172.21.15.88:0/4179066152 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf201423f0 con 0x7fcf20143d10 2024-03-29T10:47:49.832 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:49.841+0000 7fcf167fc640 10 monclient: _finish_auth 0 2024-03-29T10:47:49.832 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:49.841+0000 7fcf167fc640 10 monclient: _check_auth_tickets 2024-03-29T10:47:49.832 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:49.841+0000 7fcf167fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:47:49.832 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:49.841+0000 7fcf27d23640 5 monclient: authenticate success, global_id 4474 2024-03-29T10:47:49.832 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:49.841+0000 7fcf27d23640 10 monclient: _renew_subs 2024-03-29T10:47:49.832 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:49.841+0000 7fcf27d23640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T10:47:49.832 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:49.841+0000 7fcf27d23640 1 -- 172.21.15.88:0/4179066152 --> v1:172.21.15.196:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf201426d0 con 0x7fcf20143d10 2024-03-29T10:47:49.832 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:49.841+0000 7fcf27d23640 10 client.4474.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:47:49.832 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:49.841+0000 7fcf27d23640 10 monclient: _renew_subs 2024-03-29T10:47:49.832 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:49.841+0000 7fcf27d23640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T10:47:49.832 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:49.841+0000 7fcf27d23640 1 -- 172.21.15.88:0/4179066152 --> v1:172.21.15.196:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf20137b20 con 0x7fcf20143d10 2024-03-29T10:47:49.832 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:49.841+0000 7fcf27d23640 1 librados: init done 2024-03-29T10:47:49.832 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:49.841+0000 7fcf167fc640 1 -- 172.21.15.88:0/4179066152 <== mon.1 v1:172.21.15.196:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcf100038a0 con 0x7fcf20143d10 2024-03-29T10:47:49.832 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:49.841+0000 7fcf167fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:47:49.832 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:49.841+0000 7fcf167fc640 1 -- 172.21.15.88:0/4179066152 <== mon.1 v1:172.21.15.196:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7fcf10003cf0 con 0x7fcf20143d10 2024-03-29T10:47:49.833 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:49.841+0000 7fcf167fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:49.833 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:49.841+0000 7fcf167fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:47:49.833 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:49.841+0000 7fcf167fc640 10 monclient: dump: 2024-03-29T10:47:49.833 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:47:49.833 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:47:49.833 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:47:49.833 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:47:49.833 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:49.833 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:47:49.833 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:47:49.833 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:47:49.833 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:47:49.833 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:47:49.834 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:49.841+0000 7fcf17fff640 4 set_mon_vals no callback set 2024-03-29T10:47:49.834 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:49.843+0000 7fcf167fc640 1 -- 172.21.15.88:0/4179066152 <== mon.1 v1:172.21.15.196:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4235629220 0 0) 0x7fcf10003e50 con 0x7fcf20143d10 2024-03-29T10:47:49.834 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:49.844+0000 7fcf167fc640 1 -- 172.21.15.88:0/4179066152 <== mon.1 v1:172.21.15.196:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4257673582 0 0) 0x7fcf100839f0 con 0x7fcf20143d10 2024-03-29T10:47:49.834 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:49.844+0000 7fcf167fc640 10 client.4474.objecter ms_dispatch 0x7fcf20000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:47:49.834 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:49.844+0000 7fcf167fc640 3 client.4474.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:47:49.834 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:49.844+0000 7fcf167fc640 3 client.4474.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:47:49.834 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:49.844+0000 7fcf167fc640 20 client.4474.objecter dump_active .. 0 homeless 2024-03-29T10:47:49.834 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:49.844+0000 7fcf25abf640 10 monclient: get_auth_request con 0x7fcf0406dd90 auth_method 0 2024-03-29T10:47:49.834 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:49.845+0000 7fcf167fc640 10 client.4474.objecter ms_handle_connect 0x7fcf0406dd90 2024-03-29T10:47:49.834 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:49.845+0000 7fcf27d23640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:49.834 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:49.845+0000 7fcf27d23640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:49.835 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:49.845+0000 7fcf27d23640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T10:47:49.835 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:49.845+0000 7fcf27d23640 1 -- 172.21.15.88:0/4179066152 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf08005080 con 0x7fcf20143d10 2024-03-29T10:47:49.836 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:49.854+0000 7fcf167fc640 1 -- 172.21.15.88:0/4179066152 <== mon.1 v1:172.21.15.196:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fcf1004f2d0 con 0x7fcf20143d10 2024-03-29T10:47:49.836 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:49.854+0000 7fcf167fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:49.836 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:49.854+0000 7fcf167fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:47:50.156 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:50.173+0000 7fcf27d23640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:47:50.156 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:50.173+0000 7fcf27d23640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:47:50.156 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:50.173+0000 7fcf27d23640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T10:47:50.156 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:50.173+0000 7fcf27d23640 1 -- 172.21.15.88:0/4179066152 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcf08005330 con 0x7fcf20143d10 2024-03-29T10:47:50.157 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:50.174+0000 7fcf167fc640 1 -- 172.21.15.88:0/4179066152 <== mon.1 v1:172.21.15.196:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 344506374) 0x7fcf1004f2d0 con 0x7fcf20143d10 2024-03-29T10:47:50.157 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:50.175+0000 7fcf167fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:47:50.157 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:50.175+0000 7fcf167fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:47:50.157 INFO:teuthology.misc.health.smithi088.stdout: 2024-03-29T10:47:50.157 INFO:teuthology.misc.health.smithi088.stdout:{"epoch":10,"fsid":"869f5188-f92b-42dd-a46d-8c646e0388a9","created":"2024-03-29T10:40:40.285087+0000","modified":"2024-03-29T10:40:42.536230+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:40:42.536230+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a26c0c1c-8de0-438c-a3f6-3df10ea61574","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5de24915-8dec-49b6-8b16-f3170197c591","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ac054af3-d589-4565-9644-e6af75975c53","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3ec4f045-c714-4a22-bd1a-e07330b45ab3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"cddb9867-8e30-4951-9b13-3d1e78b9c45a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dc6d9911-785d-432f-8b53-56a1ccab302f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"77c88437-b587-496e-8b4d-1824ea8b5305","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ce963d96-5a8e-44c3-99de-c0dbebe234ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:47:50.161 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:50.178+0000 7fcf27d23640 10 librados: watch_flush enter 2024-03-29T10:47:50.161 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:50.178+0000 7fcf27d23640 10 librados: watch_flush exit 2024-03-29T10:47:50.161 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:50.178+0000 7fcf27d23640 20 client.4474.objecter shutdown clearing up homeless session... 2024-03-29T10:47:50.161 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:50.179+0000 7fcf27d23640 10 client.4474.objecter successfully canceled tick 2024-03-29T10:47:50.161 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:50.179+0000 7fcf27d23640 1 -- 172.21.15.88:0/4179066152 >> v1:172.21.15.88:6816/1752861349 conn(0x7fcf0406dd90 legacy=0x7fcf04070250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:50.161 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:50.179+0000 7fcf27d23640 10 monclient: shutdown 2024-03-29T10:47:50.161 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:50.179+0000 7fcf27d23640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:47:50.161 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:50.179+0000 7fcf27d23640 1 -- 172.21.15.88:0/4179066152 >> v1:172.21.15.196:6789/0 conn(0x7fcf20143d10 legacy=0x7fcf20154ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:50.161 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:50.179+0000 7fcf27d23640 1 -- 172.21.15.88:0/4179066152 shutdown_connections 2024-03-29T10:47:50.162 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:50.179+0000 7fcf27d23640 1 -- 172.21.15.88:0/4179066152 >> 172.21.15.88:0/4179066152 conn(0x7fcf20066a00 msgr2=0x7fcf20146ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:47:50.162 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:50.179+0000 7fcf27d23640 1 -- 172.21.15.88:0/4179066152 shutdown_connections 2024-03-29T10:47:50.162 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:50.179+0000 7fcf27d23640 1 -- 172.21.15.88:0/4179066152 wait complete. 2024-03-29T10:47:50.162 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:50.180+0000 7fcf27d23640 1 librados: shutdown 2024-03-29T10:47:50.179 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:47:56.985 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:47:57.145 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:57.161+0000 7f2a9b3f6640 10 monclient: get_monmap_and_config 2024-03-29T10:47:57.146 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:57.161+0000 7f2a9b3f6640 10 monclient: build_initial_monmap 2024-03-29T10:47:57.146 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:57.161+0000 7f2a9b3f6640 10 monclient: monmap: 2024-03-29T10:47:57.146 INFO:teuthology.misc.health.smithi088.stderr:epoch 0 2024-03-29T10:47:57.146 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:47:57.146 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:47:57.162126+0000 2024-03-29T10:47:57.146 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:47:57.162126+0000 2024-03-29T10:47:57.146 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 0 (unknown) 2024-03-29T10:47:57.146 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:47:57.146 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-03-29T10:47:57.146 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-29T10:47:57.146 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-b 2024-03-29T10:47:57.146 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:47:57.146 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:57.162+0000 7f2a9b3f6640 1 Processor -- start 2024-03-29T10:47:57.146 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:57.162+0000 7f2a9b3f6640 1 -- start start 2024-03-29T10:47:57.147 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:57.162+0000 7f2a9b3f6640 10 monclient: init 2024-03-29T10:47:57.147 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:57.163+0000 7f2a9b3f6640 10 monclient: _reopen_session rank -1 2024-03-29T10:47:57.147 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:57.163+0000 7f2a9b3f6640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:47:57.149 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:57.164+0000 7f2a9b3f6640 10 monclient(hunting): picked mon.noname-b con 0x7f2a940af6c0 addr v1:172.21.15.88:6790/0 2024-03-29T10:47:57.150 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:57.164+0000 7f2a9b3f6640 10 monclient(hunting): picked mon.noname-c con 0x7f2a9414fdc0 addr v1:172.21.15.196:6789/0 2024-03-29T10:47:57.150 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:57.164+0000 7f2a9b3f6640 10 monclient(hunting): picked mon.noname-a con 0x7f2a94153090 addr v1:172.21.15.88:6789/0 2024-03-29T10:47:57.150 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:57.164+0000 7f2a9b3f6640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a940a94e0 con 0x7f2a94153090 2024-03-29T10:47:57.150 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:57.164+0000 7f2a9b3f6640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a940af3a0 con 0x7f2a9414fdc0 2024-03-29T10:47:57.150 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:57.164+0000 7f2a9b3f6640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a940afb60 con 0x7f2a940af6c0 2024-03-29T10:47:57.150 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:57.164+0000 7f2a9b3f6640 10 monclient(hunting): _renew_subs 2024-03-29T10:47:57.150 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:57.164+0000 7f2a9b3f6640 10 monclient(hunting): authenticate will time out at 1199.400426s 2024-03-29T10:47:57.150 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:57.168+0000 7f2a99192640 1 --1- >> v1:172.21.15.88:6790/0 conn(0x7f2a940af6c0 0x7f2a9414f6b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6790/0 says I am v1:172.21.15.88:47935/0 (socket says 172.21.15.88:47935) 2024-03-29T10:47:57.150 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:57.168+0000 7f2a99192640 1 -- 172.21.15.88:0/3804187362 learned_addr learned my addr 172.21.15.88:0/3804187362 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-29T10:47:57.151 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:57.169+0000 7f2a8bfff640 1 -- 172.21.15.88:0/3804187362 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4267486025 0 0) 0x7f2a940afb60 con 0x7f2a940af6c0 2024-03-29T10:47:57.151 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:57.169+0000 7f2a8bfff640 10 monclient(hunting): _negotiate 2024-03-29T10:47:57.151 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:57.169+0000 7f2a8bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:57.152 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:57.169+0000 7f2a8bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:57.152 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:57.169+0000 7f2a8bfff640 10 monclient(hunting): my global_id is 4523 2024-03-29T10:47:57.152 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:57.169+0000 7f2a8bfff640 1 -- 172.21.15.88:0/3804187362 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2a80003520 con 0x7f2a940af6c0 2024-03-29T10:47:57.156 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:57.172+0000 7f2a8bfff640 1 -- 172.21.15.88:0/3804187362 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3557837347 0 0) 0x7f2a80003520 con 0x7f2a940af6c0 2024-03-29T10:47:57.156 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:57.172+0000 7f2a8bfff640 1 -- 172.21.15.88:0/3804187362 >> v1:172.21.15.196:6789/0 conn(0x7f2a9414fdc0 legacy=0x7f2a94152980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:57.156 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:57.172+0000 7f2a8bfff640 1 -- 172.21.15.88:0/3804187362 >> v1:172.21.15.88:6789/0 conn(0x7f2a94153090 legacy=0x7f2a9415b9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:47:57.156 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:57.172+0000 7f2a8bfff640 10 monclient: _finish_hunting 0 2024-03-29T10:47:57.156 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:57.172+0000 7f2a8bfff640 1 monclient: found mon.noname-b 2024-03-29T10:47:57.156 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:57.172+0000 7f2a8bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:47:57.156 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:57.172+0000 7f2a8bfff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.88:6790/0 2024-03-29T10:47:57.156 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:57.172+0000 7f2a8bfff640 1 -- 172.21.15.88:0/3804187362 --> v1:172.21.15.88:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a940aab80 con 0x7f2a940af6c0 2024-03-29T10:47:57.156 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:57.172+0000 7f2a8bfff640 10 monclient: _finish_auth 0 2024-03-29T10:47:57.156 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:57.172+0000 7f2a8bfff640 10 monclient: _check_auth_tickets 2024-03-29T10:47:57.156 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:57.172+0000 7f2a8bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:47:57.156 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:57.172+0000 7f2a8bfff640 1 -- 172.21.15.88:0/3804187362 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f2a90001690 con 0x7f2a940af6c0 2024-03-29T10:47:57.156 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:57.172+0000 7f2a8bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:57.157 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:57.172+0000 7f2a8bfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:47:57.157 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:57.172+0000 7f2a8bfff640 10 monclient: dump: 2024-03-29T10:47:57.157 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:47:57.157 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:47:57.157 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:47:57.157 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:47:57.157 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:57.157 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:47:57.157 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:47:57.157 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:47:57.157 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:47:57.157 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:47:57.157 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:57.172+0000 7f2a9b3f6640 5 monclient: authenticate success, global_id 4523 2024-03-29T10:47:57.157 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:57.172+0000 7f2a9b3f6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:47:57.158 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:57.176+0000 7f2a8bfff640 1 -- 172.21.15.88:0/3804187362 <== mon.2 v1:172.21.15.88:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2a90001860 con 0x7f2a940af6c0 2024-03-29T10:47:57.158 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:57.176+0000 7f2a8bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:47:57.159 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:57.176+0000 7f2a9b3f6640 10 monclient: get_monmap_and_config success 2024-03-29T10:47:57.160 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:57.176+0000 7f2a9b3f6640 4 set_mon_vals no callback set 2024-03-29T10:47:57.160 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:57.176+0000 7f2a8bfff640 1 -- 172.21.15.88:0/3804187362 <== mon.2 v1:172.21.15.88:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f2a90001c70 con 0x7f2a940af6c0 2024-03-29T10:47:57.160 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:57.176+0000 7f2a8bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:57.160 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:57.176+0000 7f2a8bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:47:57.160 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:57.176+0000 7f2a8bfff640 10 monclient: dump: 2024-03-29T10:47:57.160 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:47:57.160 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:47:57.160 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:47:57.161 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:47:57.161 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:57.161 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:47:57.161 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:47:57.161 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:47:57.161 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:47:57.161 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:47:57.161 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:57.177+0000 7f2a9b3f6640 10 monclient: shutdown 2024-03-29T10:47:57.161 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:57.177+0000 7f2a9b3f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:47:57.161 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:57.177+0000 7f2a9b3f6640 1 -- 172.21.15.88:0/3804187362 >> v1:172.21.15.88:6790/0 conn(0x7f2a940af6c0 legacy=0x7f2a9414f6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:57.161 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:57.177+0000 7f2a9b3f6640 1 -- 172.21.15.88:0/3804187362 shutdown_connections 2024-03-29T10:47:57.161 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:57.177+0000 7f2a9b3f6640 1 -- 172.21.15.88:0/3804187362 >> 172.21.15.88:0/3804187362 conn(0x7f2a940b35e0 msgr2=0x7f2a940b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:47:57.163 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:57.178+0000 7f2a9b3f6640 1 -- 172.21.15.88:0/3804187362 shutdown_connections 2024-03-29T10:47:57.163 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:57.178+0000 7f2a9b3f6640 1 -- 172.21.15.88:0/3804187362 wait complete. 2024-03-29T10:47:57.163 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:57.179+0000 7f2a9b3f6640 10 monclient: build_initial_monmap 2024-03-29T10:47:57.163 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:57.179+0000 7f2a9b3f6640 10 monclient: monmap: 2024-03-29T10:47:57.163 INFO:teuthology.misc.health.smithi088.stderr:epoch 0 2024-03-29T10:47:57.163 INFO:teuthology.misc.health.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:47:57.163 INFO:teuthology.misc.health.smithi088.stderr:last_changed 0.000000 2024-03-29T10:47:57.163 INFO:teuthology.misc.health.smithi088.stderr:created 0.000000 2024-03-29T10:47:57.163 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 0 (unknown) 2024-03-29T10:47:57.163 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:47:57.163 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-03-29T10:47:57.163 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-29T10:47:57.163 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-c 2024-03-29T10:47:57.163 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:47:57.164 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:57.179+0000 7f2a9b3f6640 1 librados: starting msgr at 2024-03-29T10:47:57.164 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:57.179+0000 7f2a9b3f6640 1 librados: starting objecter 2024-03-29T10:47:57.164 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:57.179+0000 7f2a9b3f6640 1 Processor -- start 2024-03-29T10:47:57.164 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:57.179+0000 7f2a9b3f6640 1 -- start start 2024-03-29T10:47:57.164 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:57.179+0000 7f2a9b3f6640 1 librados: setting wanted keys 2024-03-29T10:47:57.164 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:57.179+0000 7f2a9b3f6640 1 librados: calling monclient init 2024-03-29T10:47:57.164 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:57.180+0000 7f2a9b3f6640 10 monclient: init 2024-03-29T10:47:57.164 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:57.180+0000 7f2a9b3f6640 10 monclient: _reopen_session rank -1 2024-03-29T10:47:57.164 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:57.180+0000 7f2a9b3f6640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:47:57.167 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:57.183+0000 7f2a9b3f6640 10 monclient(hunting): picked mon.noname-b con 0x7f2a940af6c0 addr v1:172.21.15.196:6789/0 2024-03-29T10:47:57.167 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:57.183+0000 7f2a9b3f6640 10 monclient(hunting): picked mon.noname-a con 0x7f2a9414fdc0 addr v1:172.21.15.88:6789/0 2024-03-29T10:47:57.167 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:57.183+0000 7f2a9b3f6640 10 monclient(hunting): picked mon.noname-c con 0x7f2a94153090 addr v1:172.21.15.88:6790/0 2024-03-29T10:47:57.167 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:57.183+0000 7f2a9b3f6640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a940700c0 con 0x7f2a9414fdc0 2024-03-29T10:47:57.167 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:57.183+0000 7f2a9b3f6640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a94070310 con 0x7f2a940af6c0 2024-03-29T10:47:57.167 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:57.183+0000 7f2a9b3f6640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a94149330 con 0x7f2a94153090 2024-03-29T10:47:57.168 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:57.183+0000 7f2a9b3f6640 10 monclient(hunting): _renew_subs 2024-03-29T10:47:57.168 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:57.183+0000 7f2a9b3f6640 10 monclient(hunting): authenticate will time out at 1199.419770s 2024-03-29T10:47:57.171 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:57.187+0000 7f2a99192640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f2a940af6c0 0x7f2a94148c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.88:34723/0 (socket says 172.21.15.88:34723) 2024-03-29T10:47:57.171 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:57.187+0000 7f2a99192640 1 -- 172.21.15.88:0/2335318752 learned_addr learned my addr 172.21.15.88:0/2335318752 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-29T10:47:57.176 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:57.190+0000 7f2a89ffb640 10 client.?.objecter ms_handle_connect 0x7f2a940af6c0 2024-03-29T10:47:57.176 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:57.190+0000 7f2a89ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:47:57.176 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:57.190+0000 7f2a89ffb640 1 -- 172.21.15.88:0/2335318752 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2304202577 0 0) 0x7f2a94070310 con 0x7f2a940af6c0 2024-03-29T10:47:57.176 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:57.190+0000 7f2a89ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:47:57.176 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:57.190+0000 7f2a89ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:57.176 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:57.190+0000 7f2a89ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:57.177 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:57.190+0000 7f2a89ffb640 10 monclient(hunting): my global_id is 4477 2024-03-29T10:47:57.177 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:57.190+0000 7f2a89ffb640 1 -- 172.21.15.88:0/2335318752 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2a78004eb0 con 0x7f2a940af6c0 2024-03-29T10:47:57.177 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:57.191+0000 7f2a89ffb640 1 -- 172.21.15.88:0/2335318752 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4263522685 0 0) 0x7f2a78004eb0 con 0x7f2a940af6c0 2024-03-29T10:47:57.177 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:57.191+0000 7f2a89ffb640 1 -- 172.21.15.88:0/2335318752 --> v1:172.21.15.196:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2a94070310 con 0x7f2a940af6c0 2024-03-29T10:47:57.177 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:57.191+0000 7f2a89ffb640 1 -- 172.21.15.88:0/2335318752 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f2a90001c70 con 0x7f2a940af6c0 2024-03-29T10:47:57.177 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:57.191+0000 7f2a89ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:57.177 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:57.191+0000 7f2a89ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:47:57.177 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:57.191+0000 7f2a89ffb640 10 monclient(hunting): dump: 2024-03-29T10:47:57.177 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:47:57.177 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:47:57.177 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:47:57.178 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:47:57.178 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:57.178 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:47:57.178 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:47:57.178 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:47:57.178 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:47:57.178 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:47:57.178 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:57.191+0000 7f2a89ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:47:57.178 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:57.191+0000 7f2a89ffb640 1 -- 172.21.15.88:0/2335318752 <== mon.1 v1:172.21.15.196:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2455071312 0 0) 0x7f2a94070310 con 0x7f2a940af6c0 2024-03-29T10:47:57.178 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:57.191+0000 7f2a89ffb640 1 -- 172.21.15.88:0/2335318752 >> v1:172.21.15.88:6790/0 conn(0x7f2a94153090 legacy=0x7f2a94210ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:57.179 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:57.191+0000 7f2a89ffb640 1 -- 172.21.15.88:0/2335318752 >> v1:172.21.15.88:6789/0 conn(0x7f2a9414fdc0 legacy=0x7f2a9406e8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:57.179 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:57.191+0000 7f2a89ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:47:57.179 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:57.191+0000 7f2a89ffb640 1 monclient: found mon.b 2024-03-29T10:47:57.179 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:57.191+0000 7f2a89ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:47:57.179 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:57.191+0000 7f2a89ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T10:47:57.179 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:57.191+0000 7f2a89ffb640 1 -- 172.21.15.88:0/2335318752 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a9406efb0 con 0x7f2a940af6c0 2024-03-29T10:47:57.181 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:57.191+0000 7f2a89ffb640 10 monclient: _finish_auth 0 2024-03-29T10:47:57.181 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:57.191+0000 7f2a89ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:47:57.181 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:57.191+0000 7f2a89ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:47:57.181 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:57.192+0000 7f2a9b3f6640 5 monclient: authenticate success, global_id 4477 2024-03-29T10:47:57.181 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:57.192+0000 7f2a9b3f6640 10 monclient: _renew_subs 2024-03-29T10:47:57.181 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:57.192+0000 7f2a9b3f6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T10:47:57.181 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:57.192+0000 7f2a9b3f6640 1 -- 172.21.15.88:0/2335318752 --> v1:172.21.15.196:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a940b2ce0 con 0x7f2a940af6c0 2024-03-29T10:47:57.181 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:57.192+0000 7f2a89ffb640 1 -- 172.21.15.88:0/2335318752 <== mon.1 v1:172.21.15.196:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2a90004bf0 con 0x7f2a940af6c0 2024-03-29T10:47:57.182 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:57.192+0000 7f2a89ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:47:57.182 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:57.192+0000 7f2a89ffb640 1 -- 172.21.15.88:0/2335318752 <== mon.1 v1:172.21.15.196:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f2a90004f00 con 0x7f2a940af6c0 2024-03-29T10:47:57.182 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:57.192+0000 7f2a89ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:57.182 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:57.192+0000 7f2a89ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:47:57.182 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:57.192+0000 7f2a89ffb640 10 monclient: dump: 2024-03-29T10:47:57.182 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:47:57.182 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:47:57.182 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:47:57.182 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:47:57.182 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:57.182 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:47:57.182 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:47:57.182 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:47:57.182 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:47:57.183 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:47:57.183 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:57.192+0000 7f2a9b3f6640 10 client.4477.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:47:57.183 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:57.192+0000 7f2a9b3f6640 10 monclient: _renew_subs 2024-03-29T10:47:57.183 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:57.192+0000 7f2a9b3f6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T10:47:57.183 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:57.192+0000 7f2a9b3f6640 1 -- 172.21.15.88:0/2335318752 --> v1:172.21.15.196:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a94148660 con 0x7f2a940af6c0 2024-03-29T10:47:57.183 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:57.192+0000 7f2a9b3f6640 1 librados: init done 2024-03-29T10:47:57.184 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:57.194+0000 7f2a8b7fe640 4 set_mon_vals no callback set 2024-03-29T10:47:57.184 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:57.194+0000 7f2a89ffb640 1 -- 172.21.15.88:0/2335318752 <== mon.1 v1:172.21.15.196:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4235629220 0 0) 0x7f2a9001aed0 con 0x7f2a940af6c0 2024-03-29T10:47:57.184 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:57.196+0000 7f2a9b3f6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:57.184 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:57.196+0000 7f2a9b3f6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:57.184 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:57.196+0000 7f2a9b3f6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T10:47:57.184 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:57.196+0000 7f2a89ffb640 1 -- 172.21.15.88:0/2335318752 <== mon.1 v1:172.21.15.196:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4257673582 0 0) 0x7f2a90084c60 con 0x7f2a940af6c0 2024-03-29T10:47:57.185 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:57.196+0000 7f2a89ffb640 10 client.4477.objecter ms_dispatch 0x7f2a94000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:47:57.185 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:57.196+0000 7f2a89ffb640 3 client.4477.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:47:57.185 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:57.196+0000 7f2a89ffb640 3 client.4477.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:47:57.185 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:57.196+0000 7f2a9b3f6640 1 -- 172.21.15.88:0/2335318752 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a74005080 con 0x7f2a940af6c0 2024-03-29T10:47:57.185 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:57.196+0000 7f2a89ffb640 20 client.4477.objecter dump_active .. 0 homeless 2024-03-29T10:47:57.185 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:57.196+0000 7f2a98991640 10 monclient: get_auth_request con 0x7f2a7806dbe0 auth_method 0 2024-03-29T10:47:57.185 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:57.197+0000 7f2a89ffb640 10 client.4477.objecter ms_handle_connect 0x7f2a7806dbe0 2024-03-29T10:47:57.187 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:57.205+0000 7f2a89ffb640 1 -- 172.21.15.88:0/2335318752 <== mon.1 v1:172.21.15.196:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2a90050540 con 0x7f2a940af6c0 2024-03-29T10:47:57.187 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:57.205+0000 7f2a89ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:57.187 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:57.205+0000 7f2a89ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:47:57.555 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:57.572+0000 7f2a9b3f6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:47:57.555 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:57.572+0000 7f2a9b3f6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:47:57.555 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:57.572+0000 7f2a9b3f6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T10:47:57.555 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:57.572+0000 7f2a9b3f6640 1 -- 172.21.15.88:0/2335318752 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2a74005330 con 0x7f2a940af6c0 2024-03-29T10:47:57.556 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:57.573+0000 7f2a89ffb640 1 -- 172.21.15.88:0/2335318752 <== mon.1 v1:172.21.15.196:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 344506374) 0x7f2a90052470 con 0x7f2a940af6c0 2024-03-29T10:47:57.556 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:57.573+0000 7f2a89ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:47:57.556 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:57.573+0000 7f2a89ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:47:57.556 INFO:teuthology.misc.health.smithi088.stdout: 2024-03-29T10:47:57.556 INFO:teuthology.misc.health.smithi088.stdout:{"epoch":10,"fsid":"869f5188-f92b-42dd-a46d-8c646e0388a9","created":"2024-03-29T10:40:40.285087+0000","modified":"2024-03-29T10:40:42.536230+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:40:42.536230+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a26c0c1c-8de0-438c-a3f6-3df10ea61574","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5de24915-8dec-49b6-8b16-f3170197c591","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ac054af3-d589-4565-9644-e6af75975c53","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3ec4f045-c714-4a22-bd1a-e07330b45ab3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"cddb9867-8e30-4951-9b13-3d1e78b9c45a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dc6d9911-785d-432f-8b53-56a1ccab302f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"77c88437-b587-496e-8b4d-1824ea8b5305","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ce963d96-5a8e-44c3-99de-c0dbebe234ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:47:57.560 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:57.578+0000 7f2a9b3f6640 10 librados: watch_flush enter 2024-03-29T10:47:57.561 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:57.578+0000 7f2a9b3f6640 10 librados: watch_flush exit 2024-03-29T10:47:57.561 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:57.578+0000 7f2a9b3f6640 20 client.4477.objecter shutdown clearing up homeless session... 2024-03-29T10:47:57.561 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:57.578+0000 7f2a9b3f6640 10 client.4477.objecter successfully canceled tick 2024-03-29T10:47:57.561 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:57.578+0000 7f2a9b3f6640 1 -- 172.21.15.88:0/2335318752 >> v1:172.21.15.88:6816/1752861349 conn(0x7f2a7806dbe0 legacy=0x7f2a780700a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:57.561 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:57.578+0000 7f2a9b3f6640 10 monclient: shutdown 2024-03-29T10:47:57.561 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:57.579+0000 7f2a9b3f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:47:57.561 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:57.579+0000 7f2a9b3f6640 1 -- 172.21.15.88:0/2335318752 >> v1:172.21.15.196:6789/0 conn(0x7f2a940af6c0 legacy=0x7f2a94148c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:57.561 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:57.579+0000 7f2a9b3f6640 1 -- 172.21.15.88:0/2335318752 shutdown_connections 2024-03-29T10:47:57.561 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:57.579+0000 7f2a9b3f6640 1 -- 172.21.15.88:0/2335318752 >> 172.21.15.88:0/2335318752 conn(0x7f2a940b35e0 msgr2=0x7f2a941538a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:47:57.562 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:57.580+0000 7f2a9b3f6640 1 -- 172.21.15.88:0/2335318752 shutdown_connections 2024-03-29T10:47:57.562 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:57.580+0000 7f2a9b3f6640 1 -- 172.21.15.88:0/2335318752 wait complete. 2024-03-29T10:47:57.563 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:47:57.581+0000 7f2a9b3f6640 1 librados: shutdown 2024-03-29T10:47:57.591 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:48:04.397 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:48:04.556 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:04.572+0000 7f7af559a640 10 monclient: get_monmap_and_config 2024-03-29T10:48:04.556 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:04.572+0000 7f7af559a640 10 monclient: build_initial_monmap 2024-03-29T10:48:04.556 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:04.572+0000 7f7af559a640 10 monclient: monmap: 2024-03-29T10:48:04.556 INFO:teuthology.misc.health.smithi088.stderr:epoch 0 2024-03-29T10:48:04.556 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:48:04.556 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:48:04.574007+0000 2024-03-29T10:48:04.556 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:48:04.574007+0000 2024-03-29T10:48:04.557 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 0 (unknown) 2024-03-29T10:48:04.557 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:48:04.557 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-03-29T10:48:04.557 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-29T10:48:04.557 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-b 2024-03-29T10:48:04.557 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:48:04.557 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:04.574+0000 7f7af559a640 1 Processor -- start 2024-03-29T10:48:04.557 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:04.574+0000 7f7af559a640 1 -- start start 2024-03-29T10:48:04.557 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:04.574+0000 7f7af559a640 10 monclient: init 2024-03-29T10:48:04.560 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:04.575+0000 7f7af559a640 10 monclient: _reopen_session rank -1 2024-03-29T10:48:04.560 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:04.575+0000 7f7af559a640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:48:04.560 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:04.575+0000 7f7af559a640 10 monclient(hunting): picked mon.noname-c con 0x7f7af00b0500 addr v1:172.21.15.196:6789/0 2024-03-29T10:48:04.560 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:04.575+0000 7f7af559a640 10 monclient(hunting): picked mon.noname-a con 0x7f7af00b08e0 addr v1:172.21.15.88:6789/0 2024-03-29T10:48:04.560 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:04.575+0000 7f7af559a640 10 monclient(hunting): picked mon.noname-b con 0x7f7af00aeca0 addr v1:172.21.15.88:6790/0 2024-03-29T10:48:04.560 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:04.575+0000 7f7af559a640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7af0140620 con 0x7f7af00b08e0 2024-03-29T10:48:04.560 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:04.575+0000 7f7af559a640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7af013daf0 con 0x7f7af00b0500 2024-03-29T10:48:04.560 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:04.575+0000 7f7af559a640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7af00ae180 con 0x7f7af00aeca0 2024-03-29T10:48:04.560 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:04.575+0000 7f7af559a640 10 monclient(hunting): _renew_subs 2024-03-29T10:48:04.561 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:04.575+0000 7f7af559a640 10 monclient(hunting): authenticate will time out at 1206.811398s 2024-03-29T10:48:04.561 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:04.575+0000 7f7aeeffd640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f7af00b0500 0x7f7af013ff10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.88:35424/0 (socket says 172.21.15.88:35424) 2024-03-29T10:48:04.561 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:04.575+0000 7f7aeeffd640 1 -- 172.21.15.88:0/2047046742 learned_addr learned my addr 172.21.15.88:0/2047046742 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-29T10:48:04.561 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:04.576+0000 7f7aee7fc640 1 -- 172.21.15.88:0/2047046742 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2777472490 0 0) 0x7f7af013daf0 con 0x7f7af00b0500 2024-03-29T10:48:04.561 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:04.576+0000 7f7aee7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:48:04.561 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:04.576+0000 7f7aee7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:48:04.561 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:04.576+0000 7f7aee7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:48:04.561 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:04.576+0000 7f7aee7fc640 10 monclient(hunting): my global_id is 4480 2024-03-29T10:48:04.561 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:04.576+0000 7f7aee7fc640 1 -- 172.21.15.88:0/2047046742 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7adc003520 con 0x7f7af00b0500 2024-03-29T10:48:04.561 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:04.576+0000 7f7aee7fc640 1 -- 172.21.15.88:0/2047046742 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 941212513 0 0) 0x7f7adc003520 con 0x7f7af00b0500 2024-03-29T10:48:04.561 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:04.576+0000 7f7aee7fc640 1 -- 172.21.15.88:0/2047046742 >> v1:172.21.15.88:6790/0 conn(0x7f7af00aeca0 legacy=0x7f7af014b7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:48:04.561 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:04.576+0000 7f7aee7fc640 1 -- 172.21.15.88:0/2047046742 >> v1:172.21.15.88:6789/0 conn(0x7f7af00b08e0 legacy=0x7f7af013d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:04.561 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:04.576+0000 7f7aee7fc640 10 monclient: _finish_hunting 0 2024-03-29T10:48:04.561 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:04.576+0000 7f7aee7fc640 1 monclient: found mon.noname-c 2024-03-29T10:48:04.561 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:04.576+0000 7f7aee7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:48:04.562 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:04.576+0000 7f7aee7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.196:6789/0 2024-03-29T10:48:04.562 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:04.576+0000 7f7aee7fc640 1 -- 172.21.15.88:0/2047046742 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7af00accf0 con 0x7f7af00b0500 2024-03-29T10:48:04.562 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:04.577+0000 7f7aee7fc640 10 monclient: _finish_auth 0 2024-03-29T10:48:04.562 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:04.577+0000 7f7aee7fc640 10 monclient: _check_auth_tickets 2024-03-29T10:48:04.562 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:04.577+0000 7f7aee7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:48:04.562 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:04.577+0000 7f7af559a640 5 monclient: authenticate success, global_id 4480 2024-03-29T10:48:04.562 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:04.577+0000 7f7af559a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:48:04.562 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:04.577+0000 7f7aee7fc640 1 -- 172.21.15.88:0/2047046742 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f7ae0001690 con 0x7f7af00b0500 2024-03-29T10:48:04.562 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:04.577+0000 7f7aee7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:04.562 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:04.577+0000 7f7aee7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:48:04.562 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:04.577+0000 7f7aee7fc640 10 monclient: dump: 2024-03-29T10:48:04.562 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:48:04.562 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:48:04.562 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:48:04.563 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:48:04.563 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:04.563 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:48:04.563 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:48:04.563 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:48:04.563 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:48:04.563 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:48:04.563 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:04.577+0000 7f7af559a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:48:04.563 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:04.577+0000 7f7aee7fc640 1 -- 172.21.15.88:0/2047046742 <== mon.1 v1:172.21.15.196:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7ae0001920 con 0x7f7af00b0500 2024-03-29T10:48:04.563 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:04.577+0000 7f7aee7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:48:04.563 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:04.577+0000 7f7af559a640 10 monclient: get_monmap_and_config success 2024-03-29T10:48:04.563 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:04.577+0000 7f7af559a640 4 set_mon_vals no callback set 2024-03-29T10:48:04.563 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:04.577+0000 7f7aee7fc640 1 -- 172.21.15.88:0/2047046742 <== mon.1 v1:172.21.15.196:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f7ae0001cb0 con 0x7f7af00b0500 2024-03-29T10:48:04.563 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:04.577+0000 7f7aee7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:04.563 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:04.577+0000 7f7aee7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:48:04.564 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:04.577+0000 7f7aee7fc640 10 monclient: dump: 2024-03-29T10:48:04.564 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:48:04.564 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:48:04.564 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:48:04.564 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:48:04.564 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:04.564 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:48:04.564 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:48:04.564 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:48:04.564 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:48:04.564 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:48:04.572 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:04.578+0000 7f7af559a640 10 monclient: shutdown 2024-03-29T10:48:04.573 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:04.578+0000 7f7af559a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:48:04.573 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:04.578+0000 7f7af559a640 1 -- 172.21.15.88:0/2047046742 >> v1:172.21.15.196:6789/0 conn(0x7f7af00b0500 legacy=0x7f7af013ff10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:04.573 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:04.578+0000 7f7af559a640 1 -- 172.21.15.88:0/2047046742 shutdown_connections 2024-03-29T10:48:04.573 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:04.578+0000 7f7af559a640 1 -- 172.21.15.88:0/2047046742 >> 172.21.15.88:0/2047046742 conn(0x7f7af006b660 msgr2=0x7f7af006ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:48:04.573 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:04.579+0000 7f7af559a640 1 -- 172.21.15.88:0/2047046742 shutdown_connections 2024-03-29T10:48:04.573 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:04.579+0000 7f7af559a640 1 -- 172.21.15.88:0/2047046742 wait complete. 2024-03-29T10:48:04.573 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:04.580+0000 7f7af559a640 10 monclient: build_initial_monmap 2024-03-29T10:48:04.573 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:04.580+0000 7f7af559a640 10 monclient: monmap: 2024-03-29T10:48:04.573 INFO:teuthology.misc.health.smithi088.stderr:epoch 0 2024-03-29T10:48:04.573 INFO:teuthology.misc.health.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:48:04.573 INFO:teuthology.misc.health.smithi088.stderr:last_changed 0.000000 2024-03-29T10:48:04.574 INFO:teuthology.misc.health.smithi088.stderr:created 0.000000 2024-03-29T10:48:04.574 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 0 (unknown) 2024-03-29T10:48:04.574 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:48:04.574 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-03-29T10:48:04.574 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-29T10:48:04.574 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-c 2024-03-29T10:48:04.574 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:48:04.574 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:04.580+0000 7f7af559a640 1 librados: starting msgr at 2024-03-29T10:48:04.574 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:04.580+0000 7f7af559a640 1 librados: starting objecter 2024-03-29T10:48:04.574 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:04.580+0000 7f7af559a640 1 Processor -- start 2024-03-29T10:48:04.574 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:04.580+0000 7f7af559a640 1 -- start start 2024-03-29T10:48:04.574 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:04.580+0000 7f7af559a640 1 librados: setting wanted keys 2024-03-29T10:48:04.574 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:04.580+0000 7f7af559a640 1 librados: calling monclient init 2024-03-29T10:48:04.574 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:04.580+0000 7f7af559a640 10 monclient: init 2024-03-29T10:48:04.575 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:04.581+0000 7f7af559a640 10 monclient: _reopen_session rank -1 2024-03-29T10:48:04.575 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:04.581+0000 7f7af559a640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:48:04.575 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:04.581+0000 7f7af559a640 10 monclient(hunting): picked mon.noname-c con 0x7f7af00aeca0 addr v1:172.21.15.88:6790/0 2024-03-29T10:48:04.575 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:04.581+0000 7f7af559a640 10 monclient(hunting): picked mon.noname-b con 0x7f7af00b0500 addr v1:172.21.15.196:6789/0 2024-03-29T10:48:04.575 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:04.581+0000 7f7af559a640 10 monclient(hunting): picked mon.noname-a con 0x7f7af01e79c0 addr v1:172.21.15.88:6789/0 2024-03-29T10:48:04.575 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:04.582+0000 7f7af559a640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7af013cec0 con 0x7f7af01e79c0 2024-03-29T10:48:04.575 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:04.582+0000 7f7af559a640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7af0139050 con 0x7f7af00b0500 2024-03-29T10:48:04.575 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:04.582+0000 7f7af559a640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7af01e1750 con 0x7f7af00aeca0 2024-03-29T10:48:04.575 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:04.582+0000 7f7af559a640 10 monclient(hunting): _renew_subs 2024-03-29T10:48:04.575 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:04.582+0000 7f7af559a640 10 monclient(hunting): authenticate will time out at 1206.817983s 2024-03-29T10:48:04.575 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:04.582+0000 7f7aeeffd640 1 --1- >> v1:172.21.15.88:6790/0 conn(0x7f7af00aeca0 0x7f7af013baa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6790/0 says I am v1:172.21.15.88:47016/0 (socket says 172.21.15.88:47016) 2024-03-29T10:48:04.575 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:04.582+0000 7f7aeeffd640 1 -- 172.21.15.88:0/3132595701 learned_addr learned my addr 172.21.15.88:0/3132595701 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-29T10:48:04.575 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:04.582+0000 7f7ae7fff640 10 client.?.objecter ms_handle_connect 0x7f7af00b0500 2024-03-29T10:48:04.575 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:04.582+0000 7f7ae7fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:48:04.576 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:04.582+0000 7f7ae7fff640 10 client.?.objecter ms_handle_connect 0x7f7af01e79c0 2024-03-29T10:48:04.576 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:04.582+0000 7f7ae7fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:48:04.576 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:04.582+0000 7f7ae7fff640 1 -- 172.21.15.88:0/3132595701 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3873719745 0 0) 0x7f7af0139050 con 0x7f7af00b0500 2024-03-29T10:48:04.576 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:04.582+0000 7f7ae7fff640 10 monclient(hunting): _negotiate 2024-03-29T10:48:04.576 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:04.582+0000 7f7ae7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:48:04.576 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:04.582+0000 7f7ae7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:48:04.576 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:04.582+0000 7f7ae7fff640 10 monclient(hunting): my global_id is 4483 2024-03-29T10:48:04.576 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:04.582+0000 7f7ae7fff640 1 -- 172.21.15.88:0/3132595701 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7ad0004eb0 con 0x7f7af00b0500 2024-03-29T10:48:04.576 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:04.582+0000 7f7ae7fff640 10 client.?.objecter ms_handle_connect 0x7f7af00aeca0 2024-03-29T10:48:04.576 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:04.582+0000 7f7ae7fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:48:04.576 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:04.583+0000 7f7ae7fff640 1 -- 172.21.15.88:0/3132595701 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3386833314 0 0) 0x7f7af013cec0 con 0x7f7af01e79c0 2024-03-29T10:48:04.576 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:04.583+0000 7f7ae7fff640 10 monclient(hunting): _negotiate 2024-03-29T10:48:04.576 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:04.583+0000 7f7ae7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:48:04.576 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:04.583+0000 7f7ae7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:48:04.576 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:04.583+0000 7f7ae7fff640 10 monclient(hunting): my global_id is 4494 2024-03-29T10:48:04.577 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:04.583+0000 7f7ae7fff640 1 -- 172.21.15.88:0/3132595701 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7af0139050 con 0x7f7af01e79c0 2024-03-29T10:48:04.577 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:04.583+0000 7f7ae7fff640 1 -- 172.21.15.88:0/3132595701 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4274502785 0 0) 0x7f7af01e1750 con 0x7f7af00aeca0 2024-03-29T10:48:04.577 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:04.583+0000 7f7ae7fff640 10 monclient(hunting): _negotiate 2024-03-29T10:48:04.577 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:04.583+0000 7f7ae7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:48:04.577 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:04.583+0000 7f7ae7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:48:04.577 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:04.583+0000 7f7ae7fff640 10 monclient(hunting): my global_id is 4526 2024-03-29T10:48:04.577 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:04.583+0000 7f7ae7fff640 1 -- 172.21.15.88:0/3132595701 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7af013cec0 con 0x7f7af00aeca0 2024-03-29T10:48:04.577 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:04.583+0000 7f7ae7fff640 1 -- 172.21.15.88:0/3132595701 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2805686435 0 0) 0x7f7ad0004eb0 con 0x7f7af00b0500 2024-03-29T10:48:04.577 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:04.583+0000 7f7ae7fff640 1 -- 172.21.15.88:0/3132595701 --> v1:172.21.15.196:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7af01e1750 con 0x7f7af00b0500 2024-03-29T10:48:04.577 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:04.583+0000 7f7ae7fff640 1 -- 172.21.15.88:0/3132595701 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1751483235 0 0) 0x7f7af0139050 con 0x7f7af01e79c0 2024-03-29T10:48:04.577 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:04.583+0000 7f7ae7fff640 1 -- 172.21.15.88:0/3132595701 --> v1:172.21.15.88:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7ad0004eb0 con 0x7f7af01e79c0 2024-03-29T10:48:04.577 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:04.584+0000 7f7ae7fff640 1 -- 172.21.15.88:0/3132595701 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f7ad8001890 con 0x7f7af00b0500 2024-03-29T10:48:04.577 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:04.584+0000 7f7ae7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:04.577 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:04.584+0000 7f7ae7fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:48:04.578 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:04.584+0000 7f7ae7fff640 10 monclient(hunting): dump: 2024-03-29T10:48:04.578 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:48:04.578 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:48:04.578 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:48:04.578 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:48:04.578 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:04.578 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:48:04.578 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:48:04.578 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:48:04.578 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:48:04.578 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:48:04.578 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:04.584+0000 7f7ae7fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:48:04.578 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:04.584+0000 7f7ae7fff640 1 -- 172.21.15.88:0/3132595701 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f7ae8001810 con 0x7f7af01e79c0 2024-03-29T10:48:04.578 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:04.584+0000 7f7ae7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:04.578 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:04.584+0000 7f7ae7fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:48:04.579 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:04.584+0000 7f7ae7fff640 10 monclient(hunting): dump: 2024-03-29T10:48:04.579 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:48:04.579 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:48:04.579 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:48:04.579 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:48:04.579 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:04.579 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:48:04.579 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:48:04.579 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:48:04.579 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:48:04.579 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:48:04.579 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:04.584+0000 7f7ae7fff640 1 -- 172.21.15.88:0/3132595701 <== mon.0 v1:172.21.15.88:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3211835456 0 0) 0x7f7ad0004eb0 con 0x7f7af01e79c0 2024-03-29T10:48:04.579 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:04.584+0000 7f7ae7fff640 1 -- 172.21.15.88:0/3132595701 >> v1:172.21.15.88:6790/0 conn(0x7f7af00aeca0 legacy=0x7f7af013baa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:04.579 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:04.584+0000 7f7ae7fff640 1 -- 172.21.15.88:0/3132595701 >> v1:172.21.15.196:6789/0 conn(0x7f7af00b0500 legacy=0x7f7af013c7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:04.580 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:04.584+0000 7f7ae7fff640 10 monclient: _finish_hunting 0 2024-03-29T10:48:04.580 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:04.584+0000 7f7ae7fff640 1 monclient: found mon.a 2024-03-29T10:48:04.580 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:04.584+0000 7f7ae7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:48:04.580 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:04.584+0000 7f7ae7fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-03-29T10:48:04.580 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:04.584+0000 7f7ae7fff640 1 -- 172.21.15.88:0/3132595701 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7af01e18e0 con 0x7f7af01e79c0 2024-03-29T10:48:04.580 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:04.584+0000 7f7ae7fff640 10 monclient: _finish_auth 0 2024-03-29T10:48:04.580 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:04.584+0000 7f7ae7fff640 10 monclient: _check_auth_tickets 2024-03-29T10:48:04.580 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:04.584+0000 7f7ae7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:48:04.580 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:04.584+0000 7f7af559a640 5 monclient: authenticate success, global_id 4494 2024-03-29T10:48:04.580 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:04.584+0000 7f7af559a640 10 monclient: _renew_subs 2024-03-29T10:48:04.580 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:04.584+0000 7f7af559a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-03-29T10:48:04.580 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:04.584+0000 7f7af559a640 1 -- 172.21.15.88:0/3132595701 --> v1:172.21.15.88:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7af01e7e10 con 0x7f7af01e79c0 2024-03-29T10:48:04.580 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:04.584+0000 7f7af559a640 10 client.4494.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:48:04.580 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:04.584+0000 7f7af559a640 10 monclient: _renew_subs 2024-03-29T10:48:04.580 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:04.584+0000 7f7af559a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-03-29T10:48:04.581 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:04.584+0000 7f7af559a640 1 -- 172.21.15.88:0/3132595701 --> v1:172.21.15.88:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7af01e82a0 con 0x7f7af01e79c0 2024-03-29T10:48:04.581 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:04.584+0000 7f7af559a640 1 librados: init done 2024-03-29T10:48:04.581 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:04.586+0000 7f7ae7fff640 1 -- 172.21.15.88:0/3132595701 <== mon.0 v1:172.21.15.88:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7ae8001970 con 0x7f7af01e79c0 2024-03-29T10:48:04.581 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:04.586+0000 7f7ae7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:48:04.581 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:04.586+0000 7f7ae7fff640 1 -- 172.21.15.88:0/3132595701 <== mon.0 v1:172.21.15.88:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f7ae8001fe0 con 0x7f7af01e79c0 2024-03-29T10:48:04.581 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:04.586+0000 7f7ae7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:04.581 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:04.586+0000 7f7ae7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:48:04.581 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:04.586+0000 7f7ae7fff640 10 monclient: dump: 2024-03-29T10:48:04.581 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:48:04.581 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:48:04.581 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:48:04.581 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:48:04.581 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:04.582 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:48:04.582 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:48:04.582 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:48:04.582 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:48:04.582 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:48:04.582 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:04.587+0000 7f7aedffb640 4 set_mon_vals no callback set 2024-03-29T10:48:04.582 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:04.587+0000 7f7af559a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:48:04.582 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:04.587+0000 7f7af559a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:48:04.582 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:04.588+0000 7f7af559a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-03-29T10:48:04.582 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:04.588+0000 7f7af559a640 1 -- 172.21.15.88:0/3132595701 --> v1:172.21.15.88:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ad4005080 con 0x7f7af01e79c0 2024-03-29T10:48:04.582 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:04.589+0000 7f7ae7fff640 1 -- 172.21.15.88:0/3132595701 <== mon.0 v1:172.21.15.88:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4235629220 0 0) 0x7f7ae8019a80 con 0x7f7af01e79c0 2024-03-29T10:48:04.582 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:04.589+0000 7f7aeeffd640 10 monclient: get_auth_request con 0x7f7ad006dfc0 auth_method 0 2024-03-29T10:48:04.582 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:04.590+0000 7f7ae7fff640 1 -- 172.21.15.88:0/3132595701 <== mon.0 v1:172.21.15.88:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4257673582 0 0) 0x7f7ae8083a20 con 0x7f7af01e79c0 2024-03-29T10:48:04.583 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:04.590+0000 7f7ae7fff640 10 client.4494.objecter ms_dispatch 0x7f7af0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:48:04.583 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:04.590+0000 7f7ae7fff640 3 client.4494.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:48:04.583 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:04.590+0000 7f7ae7fff640 3 client.4494.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:48:04.583 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:04.590+0000 7f7ae7fff640 20 client.4494.objecter dump_active .. 0 homeless 2024-03-29T10:48:04.583 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:04.590+0000 7f7ae7fff640 10 client.4494.objecter ms_handle_connect 0x7f7ad006dfc0 2024-03-29T10:48:04.583 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:04.600+0000 7f7ae7fff640 1 -- 172.21.15.88:0/3132595701 <== mon.0 v1:172.21.15.88:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7ae804f300 con 0x7f7af01e79c0 2024-03-29T10:48:04.584 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:04.600+0000 7f7ae7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:48:04.584 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:04.600+0000 7f7ae7fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:48:05.016 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:05.034+0000 7f7af559a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:48:05.016 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:05.034+0000 7f7af559a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:48:05.016 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:05.034+0000 7f7af559a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-03-29T10:48:05.016 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:05.034+0000 7f7af559a640 1 -- 172.21.15.88:0/3132595701 --> v1:172.21.15.88:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7ad4005330 con 0x7f7af01e79c0 2024-03-29T10:48:05.019 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:05.036+0000 7f7ae7fff640 1 -- 172.21.15.88:0/3132595701 <== mon.0 v1:172.21.15.88:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 344506374) 0x7f7ae8051230 con 0x7f7af01e79c0 2024-03-29T10:48:05.019 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:05.037+0000 7f7ae7fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:48:05.019 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:05.037+0000 7f7ae7fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:48:05.019 INFO:teuthology.misc.health.smithi088.stdout: 2024-03-29T10:48:05.019 INFO:teuthology.misc.health.smithi088.stdout:{"epoch":10,"fsid":"869f5188-f92b-42dd-a46d-8c646e0388a9","created":"2024-03-29T10:40:40.285087+0000","modified":"2024-03-29T10:40:42.536230+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:40:42.536230+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a26c0c1c-8de0-438c-a3f6-3df10ea61574","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5de24915-8dec-49b6-8b16-f3170197c591","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ac054af3-d589-4565-9644-e6af75975c53","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3ec4f045-c714-4a22-bd1a-e07330b45ab3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"cddb9867-8e30-4951-9b13-3d1e78b9c45a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dc6d9911-785d-432f-8b53-56a1ccab302f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"77c88437-b587-496e-8b4d-1824ea8b5305","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ce963d96-5a8e-44c3-99de-c0dbebe234ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:48:05.023 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:05.040+0000 7f7af559a640 10 librados: watch_flush enter 2024-03-29T10:48:05.023 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:05.041+0000 7f7af559a640 10 librados: watch_flush exit 2024-03-29T10:48:05.023 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:05.041+0000 7f7af559a640 20 client.4494.objecter shutdown clearing up homeless session... 2024-03-29T10:48:05.023 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:05.041+0000 7f7af559a640 10 client.4494.objecter successfully canceled tick 2024-03-29T10:48:05.023 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:05.041+0000 7f7af559a640 1 -- 172.21.15.88:0/3132595701 >> v1:172.21.15.88:6816/1752861349 conn(0x7f7ad006dfc0 legacy=0x7f7ad0070480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:05.023 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:05.041+0000 7f7af559a640 10 monclient: shutdown 2024-03-29T10:48:05.024 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:05.041+0000 7f7af559a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:48:05.024 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:05.041+0000 7f7af559a640 1 -- 172.21.15.88:0/3132595701 >> v1:172.21.15.88:6789/0 conn(0x7f7af01e79c0 legacy=0x7f7af01e1040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:05.024 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:05.041+0000 7f7af559a640 1 -- 172.21.15.88:0/3132595701 shutdown_connections 2024-03-29T10:48:05.024 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:05.041+0000 7f7af559a640 1 -- 172.21.15.88:0/3132595701 >> 172.21.15.88:0/3132595701 conn(0x7f7af006b660 msgr2=0x7f7af00b2cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:48:05.024 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:05.041+0000 7f7af559a640 1 -- 172.21.15.88:0/3132595701 shutdown_connections 2024-03-29T10:48:05.024 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:05.041+0000 7f7af559a640 1 -- 172.21.15.88:0/3132595701 wait complete. 2024-03-29T10:48:05.024 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:05.041+0000 7f7af559a640 1 librados: shutdown 2024-03-29T10:48:05.040 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:48:11.847 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:48:12.006 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:12.021+0000 7f23ca30e640 10 monclient: get_monmap_and_config 2024-03-29T10:48:12.006 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:12.021+0000 7f23ca30e640 10 monclient: build_initial_monmap 2024-03-29T10:48:12.006 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:12.021+0000 7f23ca30e640 10 monclient: monmap: 2024-03-29T10:48:12.006 INFO:teuthology.misc.health.smithi088.stderr:epoch 0 2024-03-29T10:48:12.006 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:48:12.006 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:48:12.022417+0000 2024-03-29T10:48:12.006 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:48:12.022417+0000 2024-03-29T10:48:12.006 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 0 (unknown) 2024-03-29T10:48:12.006 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:48:12.006 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-03-29T10:48:12.007 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-29T10:48:12.007 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-b 2024-03-29T10:48:12.007 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:48:12.007 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:12.022+0000 7f23ca30e640 1 Processor -- start 2024-03-29T10:48:12.007 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:12.022+0000 7f23ca30e640 1 -- start start 2024-03-29T10:48:12.007 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:12.022+0000 7f23ca30e640 10 monclient: init 2024-03-29T10:48:12.007 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:12.023+0000 7f23ca30e640 10 monclient: _reopen_session rank -1 2024-03-29T10:48:12.007 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:12.023+0000 7f23ca30e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:48:12.007 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:12.023+0000 7f23ca30e640 10 monclient(hunting): picked mon.noname-a con 0x7f23c40ad6c0 addr v1:172.21.15.88:6789/0 2024-03-29T10:48:12.007 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:12.023+0000 7f23ca30e640 10 monclient(hunting): picked mon.noname-b con 0x7f23c40af6c0 addr v1:172.21.15.88:6790/0 2024-03-29T10:48:12.007 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:12.023+0000 7f23ca30e640 10 monclient(hunting): picked mon.noname-c con 0x7f23c414fdc0 addr v1:172.21.15.196:6789/0 2024-03-29T10:48:12.007 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:12.023+0000 7f23ca30e640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23c40a94e0 con 0x7f23c40ad6c0 2024-03-29T10:48:12.007 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:12.023+0000 7f23ca30e640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23c40af3a0 con 0x7f23c414fdc0 2024-03-29T10:48:12.007 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:12.023+0000 7f23ca30e640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23c40afb60 con 0x7f23c40af6c0 2024-03-29T10:48:12.008 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:12.023+0000 7f23ca30e640 10 monclient(hunting): _renew_subs 2024-03-29T10:48:12.008 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:12.023+0000 7f23ca30e640 10 monclient(hunting): authenticate will time out at 1214.259317s 2024-03-29T10:48:12.013 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:12.029+0000 7f23c3fff640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f23c40ad6c0 0x7f23c414f6b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:37825/0 (socket says 172.21.15.88:37825) 2024-03-29T10:48:12.014 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:12.029+0000 7f23c3fff640 1 -- 172.21.15.88:0/2003582687 learned_addr learned my addr 172.21.15.88:0/2003582687 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-29T10:48:12.014 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:12.029+0000 7f23c2ffd640 1 -- 172.21.15.88:0/2003582687 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2018963499 0 0) 0x7f23c40a94e0 con 0x7f23c40ad6c0 2024-03-29T10:48:12.014 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:12.029+0000 7f23c2ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:48:12.014 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:12.029+0000 7f23c2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:48:12.014 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:12.029+0000 7f23c2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:48:12.014 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:12.029+0000 7f23c2ffd640 10 monclient(hunting): my global_id is 4497 2024-03-29T10:48:12.014 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:12.029+0000 7f23c2ffd640 1 -- 172.21.15.88:0/2003582687 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f23ac003520 con 0x7f23c40ad6c0 2024-03-29T10:48:12.014 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:12.030+0000 7f23c2ffd640 1 -- 172.21.15.88:0/2003582687 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4002405206 0 0) 0x7f23ac003520 con 0x7f23c40ad6c0 2024-03-29T10:48:12.014 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:12.030+0000 7f23c2ffd640 1 -- 172.21.15.88:0/2003582687 >> v1:172.21.15.88:6790/0 conn(0x7f23c40af6c0 legacy=0x7f23c4152980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:48:12.014 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:12.030+0000 7f23c2ffd640 1 -- 172.21.15.88:0/2003582687 >> v1:172.21.15.196:6789/0 conn(0x7f23c414fdc0 legacy=0x7f23c415a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:48:12.014 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:12.030+0000 7f23c2ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:48:12.014 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:12.030+0000 7f23c2ffd640 1 monclient: found mon.noname-a 2024-03-29T10:48:12.015 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:12.030+0000 7f23c2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:48:12.015 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:12.030+0000 7f23c2ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.88:6789/0 2024-03-29T10:48:12.015 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:12.030+0000 7f23c2ffd640 1 -- 172.21.15.88:0/2003582687 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23c40aab80 con 0x7f23c40ad6c0 2024-03-29T10:48:12.015 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:12.030+0000 7f23c2ffd640 10 monclient: _finish_auth 0 2024-03-29T10:48:12.015 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:12.030+0000 7f23c2ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:48:12.015 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:12.030+0000 7f23c2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:48:12.015 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:12.030+0000 7f23c2ffd640 1 -- 172.21.15.88:0/2003582687 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f23bc001670 con 0x7f23c40ad6c0 2024-03-29T10:48:12.015 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:12.030+0000 7f23c2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:12.015 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:12.030+0000 7f23c2ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:48:12.015 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:12.030+0000 7f23c2ffd640 10 monclient: dump: 2024-03-29T10:48:12.015 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:48:12.015 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:48:12.015 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:48:12.015 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:48:12.015 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:12.016 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:48:12.016 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:48:12.016 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:48:12.016 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:48:12.016 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:48:12.016 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:12.030+0000 7f23ca30e640 5 monclient: authenticate success, global_id 4497 2024-03-29T10:48:12.016 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:12.030+0000 7f23ca30e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:48:12.016 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:12.030+0000 7f23c2ffd640 1 -- 172.21.15.88:0/2003582687 <== mon.0 v1:172.21.15.88:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f23bc001840 con 0x7f23c40ad6c0 2024-03-29T10:48:12.016 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:12.030+0000 7f23c2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:48:12.016 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:12.030+0000 7f23c2ffd640 1 -- 172.21.15.88:0/2003582687 <== mon.0 v1:172.21.15.88:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f23bc001c50 con 0x7f23c40ad6c0 2024-03-29T10:48:12.016 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:12.030+0000 7f23c2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:12.016 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:12.030+0000 7f23c2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:48:12.016 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:12.030+0000 7f23c2ffd640 10 monclient: dump: 2024-03-29T10:48:12.017 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:48:12.017 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:48:12.017 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:48:12.017 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:48:12.017 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:12.017 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:48:12.017 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:48:12.017 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:48:12.017 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:48:12.017 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:48:12.017 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:12.030+0000 7f23ca30e640 10 monclient: get_monmap_and_config success 2024-03-29T10:48:12.017 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:12.030+0000 7f23ca30e640 4 set_mon_vals no callback set 2024-03-29T10:48:12.017 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:12.031+0000 7f23ca30e640 10 monclient: shutdown 2024-03-29T10:48:12.017 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:12.031+0000 7f23ca30e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:48:12.018 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:12.031+0000 7f23ca30e640 1 -- 172.21.15.88:0/2003582687 >> v1:172.21.15.88:6789/0 conn(0x7f23c40ad6c0 legacy=0x7f23c414f6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:12.018 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:12.031+0000 7f23ca30e640 1 -- 172.21.15.88:0/2003582687 shutdown_connections 2024-03-29T10:48:12.018 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:12.031+0000 7f23ca30e640 1 -- 172.21.15.88:0/2003582687 >> 172.21.15.88:0/2003582687 conn(0x7f23c40b35e0 msgr2=0x7f23c40b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:48:12.019 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:12.031+0000 7f23ca30e640 1 -- 172.21.15.88:0/2003582687 shutdown_connections 2024-03-29T10:48:12.019 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:12.031+0000 7f23ca30e640 1 -- 172.21.15.88:0/2003582687 wait complete. 2024-03-29T10:48:12.019 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:12.032+0000 7f23ca30e640 10 monclient: build_initial_monmap 2024-03-29T10:48:12.019 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:12.032+0000 7f23ca30e640 10 monclient: monmap: 2024-03-29T10:48:12.020 INFO:teuthology.misc.health.smithi088.stderr:epoch 0 2024-03-29T10:48:12.020 INFO:teuthology.misc.health.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:48:12.020 INFO:teuthology.misc.health.smithi088.stderr:last_changed 0.000000 2024-03-29T10:48:12.020 INFO:teuthology.misc.health.smithi088.stderr:created 0.000000 2024-03-29T10:48:12.020 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 0 (unknown) 2024-03-29T10:48:12.020 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:48:12.020 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-03-29T10:48:12.020 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-29T10:48:12.020 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-c 2024-03-29T10:48:12.020 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:48:12.020 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:12.032+0000 7f23ca30e640 1 librados: starting msgr at 2024-03-29T10:48:12.020 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:12.033+0000 7f23ca30e640 1 librados: starting objecter 2024-03-29T10:48:12.020 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:12.033+0000 7f23ca30e640 1 Processor -- start 2024-03-29T10:48:12.021 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:12.033+0000 7f23ca30e640 1 -- start start 2024-03-29T10:48:12.021 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:12.033+0000 7f23ca30e640 1 librados: setting wanted keys 2024-03-29T10:48:12.021 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:12.033+0000 7f23ca30e640 1 librados: calling monclient init 2024-03-29T10:48:12.021 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:12.033+0000 7f23ca30e640 10 monclient: init 2024-03-29T10:48:12.021 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:12.034+0000 7f23ca30e640 10 monclient: _reopen_session rank -1 2024-03-29T10:48:12.021 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:12.034+0000 7f23ca30e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:48:12.021 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:12.034+0000 7f23ca30e640 10 monclient(hunting): picked mon.noname-a con 0x7f23c40ad6c0 addr v1:172.21.15.88:6789/0 2024-03-29T10:48:12.021 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:12.034+0000 7f23ca30e640 10 monclient(hunting): picked mon.noname-b con 0x7f23c40af6c0 addr v1:172.21.15.196:6789/0 2024-03-29T10:48:12.021 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:12.037+0000 7f23ca30e640 10 monclient(hunting): picked mon.noname-c con 0x7f23c414fdc0 addr v1:172.21.15.88:6790/0 2024-03-29T10:48:12.021 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:12.037+0000 7f23c88ab640 1 --1- >> v1:172.21.15.88:6790/0 conn(0x7f23c414fdc0 0x7f23c41efd60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6790/0 says I am v1:172.21.15.88:59323/0 (socket says 172.21.15.88:59323) 2024-03-29T10:48:12.021 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:12.037+0000 7f23c88ab640 1 -- 172.21.15.88:0/3051457396 learned_addr learned my addr 172.21.15.88:0/3051457396 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-29T10:48:12.021 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:12.037+0000 7f23ca30e640 1 -- 172.21.15.88:0/3051457396 --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23c41e60c0 con 0x7f23c40ad6c0 2024-03-29T10:48:12.021 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:12.037+0000 7f23c0ff9640 10 client.?.objecter ms_handle_connect 0x7f23c414fdc0 2024-03-29T10:48:12.022 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:12.037+0000 7f23c0ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:48:12.022 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:12.037+0000 7f23ca30e640 1 -- 172.21.15.88:0/3051457396 --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23c41e5490 con 0x7f23c40af6c0 2024-03-29T10:48:12.022 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:12.037+0000 7f23ca30e640 1 -- 172.21.15.88:0/3051457396 --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23c407bb30 con 0x7f23c414fdc0 2024-03-29T10:48:12.022 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:12.037+0000 7f23ca30e640 10 monclient(hunting): _renew_subs 2024-03-29T10:48:12.022 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:12.037+0000 7f23ca30e640 10 monclient(hunting): authenticate will time out at 1214.273494s 2024-03-29T10:48:12.024 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:12.038+0000 7f23c0ff9640 1 -- 172.21.15.88:0/3051457396 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3211524773 0 0) 0x7f23c407bb30 con 0x7f23c414fdc0 2024-03-29T10:48:12.024 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:12.038+0000 7f23c0ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:48:12.024 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:12.038+0000 7f23c0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:48:12.024 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:12.038+0000 7f23c0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:48:12.024 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:12.038+0000 7f23c0ff9640 10 monclient(hunting): my global_id is 4529 2024-03-29T10:48:12.024 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:12.038+0000 7f23c0ff9640 1 -- 172.21.15.88:0/3051457396 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f23b0004eb0 con 0x7f23c414fdc0 2024-03-29T10:48:12.024 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:12.039+0000 7f23c0ff9640 1 -- 172.21.15.88:0/3051457396 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3107328155 0 0) 0x7f23b0004eb0 con 0x7f23c414fdc0 2024-03-29T10:48:12.024 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:12.039+0000 7f23c0ff9640 1 -- 172.21.15.88:0/3051457396 --> v1:172.21.15.88:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f23c407bb30 con 0x7f23c414fdc0 2024-03-29T10:48:12.024 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:12.039+0000 7f23c0ff9640 1 -- 172.21.15.88:0/3051457396 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f23b4001750 con 0x7f23c414fdc0 2024-03-29T10:48:12.024 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:12.039+0000 7f23c0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:12.024 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:12.039+0000 7f23c0ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:48:12.024 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:12.039+0000 7f23c0ff9640 10 monclient(hunting): dump: 2024-03-29T10:48:12.024 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:48:12.025 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:48:12.025 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:48:12.025 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:48:12.025 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:12.025 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:48:12.025 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:48:12.025 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:48:12.025 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:48:12.025 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:48:12.025 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:12.039+0000 7f23c0ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:48:12.025 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:12.039+0000 7f23c0ff9640 1 -- 172.21.15.88:0/3051457396 <== mon.2 v1:172.21.15.88:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2619562185 0 0) 0x7f23c407bb30 con 0x7f23c414fdc0 2024-03-29T10:48:12.025 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:12.039+0000 7f23c0ff9640 1 -- 172.21.15.88:0/3051457396 >> v1:172.21.15.196:6789/0 conn(0x7f23c40af6c0 legacy=0x7f23c407c0f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:48:12.025 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:12.040+0000 7f23c0ff9640 1 -- 172.21.15.88:0/3051457396 >> v1:172.21.15.88:6789/0 conn(0x7f23c40ad6c0 legacy=0x7f23c41e59b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:48:12.025 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:12.040+0000 7f23c0ff9640 10 monclient: _finish_hunting 0 2024-03-29T10:48:12.026 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:12.040+0000 7f23c0ff9640 1 monclient: found mon.c 2024-03-29T10:48:12.026 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:12.040+0000 7f23c0ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:48:12.026 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:12.040+0000 7f23c0ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-03-29T10:48:12.026 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:12.040+0000 7f23c0ff9640 1 -- 172.21.15.88:0/3051457396 --> v1:172.21.15.88:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23c407b810 con 0x7f23c414fdc0 2024-03-29T10:48:12.026 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:12.040+0000 7f23c0ff9640 10 monclient: _finish_auth 0 2024-03-29T10:48:12.026 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:12.040+0000 7f23c0ff9640 10 monclient: _check_auth_tickets 2024-03-29T10:48:12.026 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:12.040+0000 7f23c0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:48:12.026 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:12.040+0000 7f23ca30e640 5 monclient: authenticate success, global_id 4529 2024-03-29T10:48:12.026 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:12.040+0000 7f23ca30e640 10 monclient: _renew_subs 2024-03-29T10:48:12.026 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:12.040+0000 7f23ca30e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-03-29T10:48:12.026 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:12.040+0000 7f23ca30e640 1 -- 172.21.15.88:0/3051457396 --> v1:172.21.15.88:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23c407c800 con 0x7f23c414fdc0 2024-03-29T10:48:12.026 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:12.040+0000 7f23c0ff9640 1 -- 172.21.15.88:0/3051457396 <== mon.2 v1:172.21.15.88:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f23b4001d20 con 0x7f23c414fdc0 2024-03-29T10:48:12.026 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:12.040+0000 7f23c0ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:48:12.027 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:12.040+0000 7f23c0ff9640 1 -- 172.21.15.88:0/3051457396 <== mon.2 v1:172.21.15.88:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f23b4004a00 con 0x7f23c414fdc0 2024-03-29T10:48:12.027 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:12.040+0000 7f23c0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:12.027 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:12.040+0000 7f23c0ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:48:12.027 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:12.040+0000 7f23c0ff9640 10 monclient: dump: 2024-03-29T10:48:12.027 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:48:12.027 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:48:12.027 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:48:12.027 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:48:12.027 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:12.027 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:48:12.027 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:48:12.027 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:48:12.027 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:48:12.027 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:48:12.028 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:12.040+0000 7f23ca30e640 10 client.4529.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:48:12.028 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:12.040+0000 7f23ca30e640 10 monclient: _renew_subs 2024-03-29T10:48:12.028 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:12.040+0000 7f23ca30e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-03-29T10:48:12.028 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:12.040+0000 7f23ca30e640 1 -- 172.21.15.88:0/3051457396 --> v1:172.21.15.88:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f23c4070380 con 0x7f23c414fdc0 2024-03-29T10:48:12.028 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:12.040+0000 7f23ca30e640 1 librados: init done 2024-03-29T10:48:12.028 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:12.041+0000 7f23c27fc640 4 set_mon_vals no callback set 2024-03-29T10:48:12.029 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:12.043+0000 7f23c0ff9640 1 -- 172.21.15.88:0/3051457396 <== mon.2 v1:172.21.15.88:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4235629220 0 0) 0x7f23b401a9d0 con 0x7f23c414fdc0 2024-03-29T10:48:12.029 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:12.043+0000 7f23c0ff9640 1 -- 172.21.15.88:0/3051457396 <== mon.2 v1:172.21.15.88:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4257673582 0 0) 0x7f23b40547f0 con 0x7f23c414fdc0 2024-03-29T10:48:12.029 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:12.043+0000 7f23c0ff9640 10 client.4529.objecter ms_dispatch 0x7f23c4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:48:12.029 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:12.043+0000 7f23c0ff9640 3 client.4529.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:48:12.029 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:12.043+0000 7f23c0ff9640 3 client.4529.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:48:12.029 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:12.043+0000 7f23c0ff9640 20 client.4529.objecter dump_active .. 0 homeless 2024-03-29T10:48:12.029 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:12.044+0000 7f23c3fff640 10 monclient: get_auth_request con 0x7f23b006fcd0 auth_method 0 2024-03-29T10:48:12.029 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:12.044+0000 7f23c0ff9640 10 client.4529.objecter ms_handle_connect 0x7f23b006fcd0 2024-03-29T10:48:12.030 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:12.044+0000 7f23ca30e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:48:12.030 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:12.044+0000 7f23ca30e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:48:12.030 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:12.044+0000 7f23ca30e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-03-29T10:48:12.030 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:12.044+0000 7f23ca30e640 1 -- 172.21.15.88:0/3051457396 --> v1:172.21.15.88:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23a4005080 con 0x7f23c414fdc0 2024-03-29T10:48:12.039 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:12.057+0000 7f23c0ff9640 1 -- 172.21.15.88:0/3051457396 <== mon.2 v1:172.21.15.88:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f23b4050a90 con 0x7f23c414fdc0 2024-03-29T10:48:12.039 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:12.057+0000 7f23c0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:48:12.039 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:12.057+0000 7f23c0ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:48:12.392 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:12.410+0000 7f23ca30e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:48:12.392 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:12.410+0000 7f23ca30e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:48:12.392 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:12.410+0000 7f23ca30e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-03-29T10:48:12.392 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:12.410+0000 7f23ca30e640 1 -- 172.21.15.88:0/3051457396 --> v1:172.21.15.88:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f23a4005330 con 0x7f23c414fdc0 2024-03-29T10:48:12.394 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:12.411+0000 7f23c0ff9640 1 -- 172.21.15.88:0/3051457396 <== mon.2 v1:172.21.15.88:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 344506374) 0x7f23b4050c30 con 0x7f23c414fdc0 2024-03-29T10:48:12.394 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:12.411+0000 7f23c0ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:48:12.394 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:12.411+0000 7f23c0ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:48:12.394 INFO:teuthology.misc.health.smithi088.stdout: 2024-03-29T10:48:12.394 INFO:teuthology.misc.health.smithi088.stdout:{"epoch":10,"fsid":"869f5188-f92b-42dd-a46d-8c646e0388a9","created":"2024-03-29T10:40:40.285087+0000","modified":"2024-03-29T10:40:42.536230+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:40:42.536230+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a26c0c1c-8de0-438c-a3f6-3df10ea61574","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5de24915-8dec-49b6-8b16-f3170197c591","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ac054af3-d589-4565-9644-e6af75975c53","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3ec4f045-c714-4a22-bd1a-e07330b45ab3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"cddb9867-8e30-4951-9b13-3d1e78b9c45a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dc6d9911-785d-432f-8b53-56a1ccab302f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"77c88437-b587-496e-8b4d-1824ea8b5305","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ce963d96-5a8e-44c3-99de-c0dbebe234ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:48:12.398 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:12.415+0000 7f23ca30e640 10 librados: watch_flush enter 2024-03-29T10:48:12.398 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:12.415+0000 7f23ca30e640 10 librados: watch_flush exit 2024-03-29T10:48:12.398 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:12.415+0000 7f23ca30e640 20 client.4529.objecter shutdown clearing up homeless session... 2024-03-29T10:48:12.398 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:12.415+0000 7f23ca30e640 10 client.4529.objecter successfully canceled tick 2024-03-29T10:48:12.398 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:12.415+0000 7f23ca30e640 1 -- 172.21.15.88:0/3051457396 >> v1:172.21.15.88:6816/1752861349 conn(0x7f23b006fcd0 legacy=0x7f23b0072190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:12.398 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:12.415+0000 7f23ca30e640 10 monclient: shutdown 2024-03-29T10:48:12.398 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:12.415+0000 7f23ca30e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:48:12.398 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:12.415+0000 7f23ca30e640 1 -- 172.21.15.88:0/3051457396 >> v1:172.21.15.88:6790/0 conn(0x7f23c414fdc0 legacy=0x7f23c41efd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:12.398 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:12.415+0000 7f23ca30e640 1 -- 172.21.15.88:0/3051457396 shutdown_connections 2024-03-29T10:48:12.398 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:12.415+0000 7f23ca30e640 1 -- 172.21.15.88:0/3051457396 >> 172.21.15.88:0/3051457396 conn(0x7f23c40b35e0 msgr2=0x7f23c40a94e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:48:12.398 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:12.415+0000 7f23ca30e640 1 -- 172.21.15.88:0/3051457396 shutdown_connections 2024-03-29T10:48:12.398 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:12.416+0000 7f23ca30e640 1 -- 172.21.15.88:0/3051457396 wait complete. 2024-03-29T10:48:12.399 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:12.416+0000 7f23ca30e640 1 librados: shutdown 2024-03-29T10:48:12.413 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:48:19.221 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:48:19.371 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:19.389+0000 7f13ef08d640 10 monclient: get_monmap_and_config 2024-03-29T10:48:19.371 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:19.389+0000 7f13ef08d640 10 monclient: build_initial_monmap 2024-03-29T10:48:19.371 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:19.389+0000 7f13ef08d640 10 monclient: monmap: 2024-03-29T10:48:19.371 INFO:teuthology.misc.health.smithi088.stderr:epoch 0 2024-03-29T10:48:19.371 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:48:19.372 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:48:19.390214+0000 2024-03-29T10:48:19.372 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:48:19.390214+0000 2024-03-29T10:48:19.372 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 0 (unknown) 2024-03-29T10:48:19.372 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:48:19.372 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-03-29T10:48:19.372 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-29T10:48:19.372 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-b 2024-03-29T10:48:19.372 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:48:19.373 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:19.390+0000 7f13ef08d640 1 Processor -- start 2024-03-29T10:48:19.373 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:19.390+0000 7f13ef08d640 1 -- start start 2024-03-29T10:48:19.373 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:19.391+0000 7f13ef08d640 10 monclient: init 2024-03-29T10:48:19.373 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:19.391+0000 7f13ef08d640 10 monclient: _reopen_session rank -1 2024-03-29T10:48:19.373 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:19.391+0000 7f13ef08d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:48:19.374 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:19.391+0000 7f13ef08d640 10 monclient(hunting): picked mon.noname-b con 0x7f13e8071de0 addr v1:172.21.15.88:6790/0 2024-03-29T10:48:19.374 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:19.391+0000 7f13ef08d640 10 monclient(hunting): picked mon.noname-c con 0x7f13e80721c0 addr v1:172.21.15.196:6789/0 2024-03-29T10:48:19.374 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:19.391+0000 7f13ef08d640 10 monclient(hunting): picked mon.noname-a con 0x7f13e8145ae0 addr v1:172.21.15.88:6789/0 2024-03-29T10:48:19.374 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:19.391+0000 7f13ef08d640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13e8145650 con 0x7f13e8145ae0 2024-03-29T10:48:19.374 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:19.391+0000 7f13ef08d640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13e80ae260 con 0x7f13e80721c0 2024-03-29T10:48:19.374 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:19.391+0000 7f13ef08d640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13e81420d0 con 0x7f13e8071de0 2024-03-29T10:48:19.374 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:19.391+0000 7f13ef08d640 10 monclient(hunting): _renew_subs 2024-03-29T10:48:19.374 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:19.391+0000 7f13ef08d640 10 monclient(hunting): authenticate will time out at 1221.627781s 2024-03-29T10:48:19.377 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:19.395+0000 7f13e7fff640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f13e80721c0 0x7f13e813f400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.88:39453/0 (socket says 172.21.15.88:39453) 2024-03-29T10:48:19.377 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:19.395+0000 7f13e7fff640 1 -- 172.21.15.88:0/517094880 learned_addr learned my addr 172.21.15.88:0/517094880 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-29T10:48:19.380 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:19.398+0000 7f13e77fe640 1 -- 172.21.15.88:0/517094880 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4147739204 0 0) 0x7f13e80ae260 con 0x7f13e80721c0 2024-03-29T10:48:19.380 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:19.398+0000 7f13e77fe640 10 monclient(hunting): _negotiate 2024-03-29T10:48:19.380 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:19.398+0000 7f13e77fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:48:19.380 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:19.398+0000 7f13e77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:48:19.380 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:19.398+0000 7f13e77fe640 10 monclient(hunting): my global_id is 4486 2024-03-29T10:48:19.381 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:19.398+0000 7f13e77fe640 1 -- 172.21.15.88:0/517094880 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f13d0003520 con 0x7f13e80721c0 2024-03-29T10:48:19.381 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:19.398+0000 7f13e77fe640 1 -- 172.21.15.88:0/517094880 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 271744706 0 0) 0x7f13d0003520 con 0x7f13e80721c0 2024-03-29T10:48:19.381 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:19.398+0000 7f13e77fe640 1 -- 172.21.15.88:0/517094880 >> v1:172.21.15.88:6790/0 conn(0x7f13e8071de0 legacy=0x7f13e80aed80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:48:19.381 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:19.398+0000 7f13e77fe640 1 -- 172.21.15.88:0/517094880 >> v1:172.21.15.88:6789/0 conn(0x7f13e8145ae0 legacy=0x7f13e8145f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:48:19.381 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:19.398+0000 7f13e77fe640 10 monclient: _finish_hunting 0 2024-03-29T10:48:19.381 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:19.398+0000 7f13e77fe640 1 monclient: found mon.noname-c 2024-03-29T10:48:19.382 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:19.398+0000 7f13e77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:48:19.382 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:19.398+0000 7f13e77fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.196:6789/0 2024-03-29T10:48:19.382 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:19.398+0000 7f13e77fe640 1 -- 172.21.15.88:0/517094880 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13e80b0590 con 0x7f13e80721c0 2024-03-29T10:48:19.382 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:19.398+0000 7f13e77fe640 10 monclient: _finish_auth 0 2024-03-29T10:48:19.382 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:19.398+0000 7f13e77fe640 10 monclient: _check_auth_tickets 2024-03-29T10:48:19.382 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:19.398+0000 7f13e77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:48:19.382 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:19.398+0000 7f13e77fe640 1 -- 172.21.15.88:0/517094880 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f13d8001690 con 0x7f13e80721c0 2024-03-29T10:48:19.382 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:19.398+0000 7f13e77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:19.382 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:19.398+0000 7f13e77fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:48:19.382 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:19.398+0000 7f13e77fe640 10 monclient: dump: 2024-03-29T10:48:19.382 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:48:19.382 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:48:19.382 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:48:19.383 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:48:19.383 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:19.383 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:48:19.383 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:48:19.383 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:48:19.383 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:48:19.383 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:48:19.383 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:19.399+0000 7f13e77fe640 1 -- 172.21.15.88:0/517094880 <== mon.1 v1:172.21.15.196:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f13d80017f0 con 0x7f13e80721c0 2024-03-29T10:48:19.383 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:19.399+0000 7f13e77fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:48:19.383 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:19.399+0000 7f13e77fe640 1 -- 172.21.15.88:0/517094880 <== mon.1 v1:172.21.15.196:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f13d8001c40 con 0x7f13e80721c0 2024-03-29T10:48:19.383 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:19.399+0000 7f13e77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:19.383 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:19.399+0000 7f13e77fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:48:19.383 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:19.399+0000 7f13e77fe640 10 monclient: dump: 2024-03-29T10:48:19.383 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:48:19.383 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:48:19.384 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:48:19.384 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:48:19.384 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:19.384 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:48:19.384 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:48:19.384 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:48:19.384 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:48:19.384 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:48:19.385 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:19.399+0000 7f13ef08d640 5 monclient: authenticate success, global_id 4486 2024-03-29T10:48:19.385 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:19.399+0000 7f13ef08d640 10 monclient: get_monmap_and_config success 2024-03-29T10:48:19.385 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:19.399+0000 7f13ef08d640 4 set_mon_vals no callback set 2024-03-29T10:48:19.385 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:19.400+0000 7f13ef08d640 10 monclient: shutdown 2024-03-29T10:48:19.385 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:19.400+0000 7f13ef08d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:48:19.385 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:19.400+0000 7f13ef08d640 1 -- 172.21.15.88:0/517094880 >> v1:172.21.15.196:6789/0 conn(0x7f13e80721c0 legacy=0x7f13e813f400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:19.389 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:19.406+0000 7f13ef08d640 1 -- 172.21.15.88:0/517094880 shutdown_connections 2024-03-29T10:48:19.389 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:19.406+0000 7f13ef08d640 1 -- 172.21.15.88:0/517094880 >> 172.21.15.88:0/517094880 conn(0x7f13e806b660 msgr2=0x7f13e806ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:48:19.389 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:19.407+0000 7f13ef08d640 1 -- 172.21.15.88:0/517094880 shutdown_connections 2024-03-29T10:48:19.389 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:19.407+0000 7f13ef08d640 1 -- 172.21.15.88:0/517094880 wait complete. 2024-03-29T10:48:19.391 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:19.408+0000 7f13ef08d640 10 monclient: build_initial_monmap 2024-03-29T10:48:19.392 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:19.408+0000 7f13ef08d640 10 monclient: monmap: 2024-03-29T10:48:19.392 INFO:teuthology.misc.health.smithi088.stderr:epoch 0 2024-03-29T10:48:19.392 INFO:teuthology.misc.health.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:48:19.392 INFO:teuthology.misc.health.smithi088.stderr:last_changed 0.000000 2024-03-29T10:48:19.392 INFO:teuthology.misc.health.smithi088.stderr:created 0.000000 2024-03-29T10:48:19.392 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 0 (unknown) 2024-03-29T10:48:19.392 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:48:19.392 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-03-29T10:48:19.392 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-29T10:48:19.392 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-c 2024-03-29T10:48:19.392 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:48:19.392 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:19.409+0000 7f13ef08d640 1 librados: starting msgr at 2024-03-29T10:48:19.392 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:19.409+0000 7f13ef08d640 1 librados: starting objecter 2024-03-29T10:48:19.392 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:19.409+0000 7f13ef08d640 1 Processor -- start 2024-03-29T10:48:19.393 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:19.409+0000 7f13ef08d640 1 -- start start 2024-03-29T10:48:19.393 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:19.409+0000 7f13ef08d640 1 librados: setting wanted keys 2024-03-29T10:48:19.393 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:19.409+0000 7f13ef08d640 1 librados: calling monclient init 2024-03-29T10:48:19.393 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:19.409+0000 7f13ef08d640 10 monclient: init 2024-03-29T10:48:19.393 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:19.410+0000 7f13ef08d640 10 monclient: _reopen_session rank -1 2024-03-29T10:48:19.393 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:19.410+0000 7f13ef08d640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:48:19.393 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:19.410+0000 7f13ef08d640 10 monclient(hunting): picked mon.noname-b con 0x7f13e8071de0 addr v1:172.21.15.196:6789/0 2024-03-29T10:48:19.393 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:19.410+0000 7f13ef08d640 10 monclient(hunting): picked mon.noname-c con 0x7f13e8145ae0 addr v1:172.21.15.88:6790/0 2024-03-29T10:48:19.393 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:19.410+0000 7f13ef08d640 10 monclient(hunting): picked mon.noname-a con 0x7f13e8132780 addr v1:172.21.15.88:6789/0 2024-03-29T10:48:19.393 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:19.410+0000 7f13ef08d640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13e8131810 con 0x7f13e8132780 2024-03-29T10:48:19.394 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:19.410+0000 7f13ef08d640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13e81d70b0 con 0x7f13e8071de0 2024-03-29T10:48:19.394 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:19.410+0000 7f13ef08d640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13e81d7f30 con 0x7f13e8145ae0 2024-03-29T10:48:19.394 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:19.410+0000 7f13ef08d640 10 monclient(hunting): _renew_subs 2024-03-29T10:48:19.394 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:19.410+0000 7f13ef08d640 10 monclient(hunting): authenticate will time out at 1221.646381s 2024-03-29T10:48:19.397 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:19.415+0000 7f13ed62a640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f13e8132780 0x7f13e81e18e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:60303/0 (socket says 172.21.15.88:60303) 2024-03-29T10:48:19.397 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:19.415+0000 7f13ed62a640 1 -- 172.21.15.88:0/2450248947 learned_addr learned my addr 172.21.15.88:0/2450248947 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-29T10:48:19.397 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:19.415+0000 7f13e57fa640 10 client.?.objecter ms_handle_connect 0x7f13e8132780 2024-03-29T10:48:19.397 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:19.415+0000 7f13e57fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:48:19.397 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:19.415+0000 7f13e57fa640 1 -- 172.21.15.88:0/2450248947 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2660059705 0 0) 0x7f13e8131810 con 0x7f13e8132780 2024-03-29T10:48:19.397 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:19.415+0000 7f13e57fa640 10 monclient(hunting): _negotiate 2024-03-29T10:48:19.398 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:19.415+0000 7f13e57fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:48:19.398 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:19.415+0000 7f13e57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:48:19.398 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:19.415+0000 7f13e57fa640 10 monclient(hunting): my global_id is 4500 2024-03-29T10:48:19.398 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:19.415+0000 7f13e57fa640 1 -- 172.21.15.88:0/2450248947 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f13d4004eb0 con 0x7f13e8132780 2024-03-29T10:48:19.398 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:19.416+0000 7f13e57fa640 1 -- 172.21.15.88:0/2450248947 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2443857513 0 0) 0x7f13d4004eb0 con 0x7f13e8132780 2024-03-29T10:48:19.399 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:19.416+0000 7f13e57fa640 1 -- 172.21.15.88:0/2450248947 --> v1:172.21.15.88:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f13d4005300 con 0x7f13e8132780 2024-03-29T10:48:19.399 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:19.416+0000 7f13e57fa640 1 -- 172.21.15.88:0/2450248947 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f13dc003fc0 con 0x7f13e8132780 2024-03-29T10:48:19.399 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:19.416+0000 7f13e57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:19.399 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:19.416+0000 7f13e57fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:48:19.399 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:19.416+0000 7f13e57fa640 10 monclient(hunting): dump: 2024-03-29T10:48:19.399 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:48:19.399 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:48:19.399 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:48:19.399 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:48:19.399 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:19.399 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:48:19.399 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:48:19.399 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:48:19.400 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:48:19.400 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:48:19.400 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:19.416+0000 7f13e57fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:48:19.400 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:19.416+0000 7f13e57fa640 1 -- 172.21.15.88:0/2450248947 <== mon.0 v1:172.21.15.88:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2071952491 0 0) 0x7f13d4005300 con 0x7f13e8132780 2024-03-29T10:48:19.400 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:19.416+0000 7f13e57fa640 1 -- 172.21.15.88:0/2450248947 >> v1:172.21.15.88:6790/0 conn(0x7f13e8145ae0 legacy=0x7f13e8132e40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:48:19.400 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:19.417+0000 7f13e57fa640 1 -- 172.21.15.88:0/2450248947 >> v1:172.21.15.196:6789/0 conn(0x7f13e8071de0 legacy=0x7f13e81d7820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:19.400 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:19.417+0000 7f13e57fa640 10 monclient: _finish_hunting 0 2024-03-29T10:48:19.400 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:19.417+0000 7f13e57fa640 1 monclient: found mon.a 2024-03-29T10:48:19.400 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:19.417+0000 7f13e57fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:48:19.400 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:19.417+0000 7f13e57fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-03-29T10:48:19.401 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:19.417+0000 7f13e57fa640 1 -- 172.21.15.88:0/2450248947 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13e81d7300 con 0x7f13e8132780 2024-03-29T10:48:19.401 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:19.417+0000 7f13e57fa640 10 monclient: _finish_auth 0 2024-03-29T10:48:19.401 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:19.417+0000 7f13e57fa640 10 monclient: _check_auth_tickets 2024-03-29T10:48:19.401 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:19.417+0000 7f13e57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:48:19.401 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:19.417+0000 7f13e57fa640 10 client.?.objecter ms_handle_connect 0x7f13e8145ae0 2024-03-29T10:48:19.401 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:19.417+0000 7f13e57fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:48:19.401 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:19.417+0000 7f13ef08d640 5 monclient: authenticate success, global_id 4500 2024-03-29T10:48:19.401 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:19.417+0000 7f13ef08d640 10 monclient: _renew_subs 2024-03-29T10:48:19.401 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:19.417+0000 7f13ef08d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-03-29T10:48:19.401 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:19.417+0000 7f13ef08d640 1 -- 172.21.15.88:0/2450248947 --> v1:172.21.15.88:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13e81d4c30 con 0x7f13e8132780 2024-03-29T10:48:19.401 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:19.417+0000 7f13ef08d640 10 client.4500.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:48:19.401 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:19.417+0000 7f13ef08d640 10 monclient: _renew_subs 2024-03-29T10:48:19.401 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:19.417+0000 7f13ef08d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-03-29T10:48:19.402 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:19.417+0000 7f13ef08d640 1 -- 172.21.15.88:0/2450248947 --> v1:172.21.15.88:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f13e81e3000 con 0x7f13e8132780 2024-03-29T10:48:19.402 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:19.417+0000 7f13ef08d640 1 librados: init done 2024-03-29T10:48:19.405 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:19.421+0000 7f13e57fa640 1 -- 172.21.15.88:0/2450248947 <== mon.0 v1:172.21.15.88:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f13dc000ed0 con 0x7f13e8132780 2024-03-29T10:48:19.405 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:19.421+0000 7f13e57fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:48:19.405 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:19.421+0000 7f13e6ffd640 4 set_mon_vals no callback set 2024-03-29T10:48:19.405 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:19.422+0000 7f13e57fa640 1 -- 172.21.15.88:0/2450248947 <== mon.0 v1:172.21.15.88:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f13dc003fc0 con 0x7f13e8132780 2024-03-29T10:48:19.406 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:19.422+0000 7f13e57fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:19.406 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:19.422+0000 7f13e57fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:48:19.406 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:19.422+0000 7f13e57fa640 10 monclient: dump: 2024-03-29T10:48:19.406 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:48:19.406 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:48:19.406 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:48:19.406 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:48:19.406 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:19.406 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:48:19.406 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:48:19.406 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:48:19.406 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:48:19.406 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:48:19.407 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:19.423+0000 7f13ef08d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:48:19.407 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:19.423+0000 7f13ef08d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:48:19.407 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:19.423+0000 7f13ef08d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-03-29T10:48:19.407 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:19.423+0000 7f13ef08d640 1 -- 172.21.15.88:0/2450248947 --> v1:172.21.15.88:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13c8005080 con 0x7f13e8132780 2024-03-29T10:48:19.408 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:19.423+0000 7f13e57fa640 1 -- 172.21.15.88:0/2450248947 <== mon.0 v1:172.21.15.88:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4235629220 0 0) 0x7f13dc001200 con 0x7f13e8132780 2024-03-29T10:48:19.408 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:19.424+0000 7f13e57fa640 1 -- 172.21.15.88:0/2450248947 <== mon.0 v1:172.21.15.88:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4257673582 0 0) 0x7f13dc0847d0 con 0x7f13e8132780 2024-03-29T10:48:19.408 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:19.424+0000 7f13e57fa640 10 client.4500.objecter ms_dispatch 0x7f13e8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:48:19.408 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:19.424+0000 7f13e57fa640 3 client.4500.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:48:19.408 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:19.424+0000 7f13e57fa640 3 client.4500.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:48:19.408 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:19.424+0000 7f13e57fa640 20 client.4500.objecter dump_active .. 0 homeless 2024-03-29T10:48:19.408 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:19.424+0000 7f13ece29640 10 monclient: get_auth_request con 0x7f13d406df90 auth_method 0 2024-03-29T10:48:19.408 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:19.424+0000 7f13e57fa640 10 client.4500.objecter ms_handle_connect 0x7f13d406df90 2024-03-29T10:48:19.413 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:19.431+0000 7f13e57fa640 1 -- 172.21.15.88:0/2450248947 <== mon.0 v1:172.21.15.88:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f13dc0501d0 con 0x7f13e8132780 2024-03-29T10:48:19.413 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:19.431+0000 7f13e57fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:48:19.413 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:19.431+0000 7f13e57fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:48:19.742 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:19.759+0000 7f13ef08d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:48:19.743 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:19.759+0000 7f13ef08d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:48:19.743 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:19.759+0000 7f13ef08d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-03-29T10:48:19.743 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:19.759+0000 7f13ef08d640 1 -- 172.21.15.88:0/2450248947 --> v1:172.21.15.88:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f13c8005330 con 0x7f13e8132780 2024-03-29T10:48:19.743 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:19.760+0000 7f13e57fa640 1 -- 172.21.15.88:0/2450248947 <== mon.0 v1:172.21.15.88:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 344506374) 0x7f13dc0501d0 con 0x7f13e8132780 2024-03-29T10:48:19.743 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:19.760+0000 7f13e57fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:48:19.743 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:19.760+0000 7f13e57fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:48:19.744 INFO:teuthology.misc.health.smithi088.stdout: 2024-03-29T10:48:19.744 INFO:teuthology.misc.health.smithi088.stdout:{"epoch":10,"fsid":"869f5188-f92b-42dd-a46d-8c646e0388a9","created":"2024-03-29T10:40:40.285087+0000","modified":"2024-03-29T10:40:42.536230+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:40:42.536230+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a26c0c1c-8de0-438c-a3f6-3df10ea61574","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5de24915-8dec-49b6-8b16-f3170197c591","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ac054af3-d589-4565-9644-e6af75975c53","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3ec4f045-c714-4a22-bd1a-e07330b45ab3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"cddb9867-8e30-4951-9b13-3d1e78b9c45a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dc6d9911-785d-432f-8b53-56a1ccab302f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"77c88437-b587-496e-8b4d-1824ea8b5305","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ce963d96-5a8e-44c3-99de-c0dbebe234ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:48:19.746 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:19.763+0000 7f13ef08d640 10 librados: watch_flush enter 2024-03-29T10:48:19.746 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:19.764+0000 7f13ef08d640 10 librados: watch_flush exit 2024-03-29T10:48:19.746 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:19.764+0000 7f13ef08d640 20 client.4500.objecter shutdown clearing up homeless session... 2024-03-29T10:48:19.746 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:19.764+0000 7f13ef08d640 10 client.4500.objecter successfully canceled tick 2024-03-29T10:48:19.750 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:19.767+0000 7f13ef08d640 1 -- 172.21.15.88:0/2450248947 >> v1:172.21.15.88:6816/1752861349 conn(0x7f13d406df90 legacy=0x7f13d4070450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:19.750 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:19.767+0000 7f13ef08d640 10 monclient: shutdown 2024-03-29T10:48:19.750 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:19.767+0000 7f13ef08d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:48:19.750 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:19.767+0000 7f13ef08d640 1 -- 172.21.15.88:0/2450248947 >> v1:172.21.15.88:6789/0 conn(0x7f13e8132780 legacy=0x7f13e81e18e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:19.750 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:19.767+0000 7f13ef08d640 1 -- 172.21.15.88:0/2450248947 shutdown_connections 2024-03-29T10:48:19.751 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:19.767+0000 7f13ef08d640 1 -- 172.21.15.88:0/2450248947 >> 172.21.15.88:0/2450248947 conn(0x7f13e806b660 msgr2=0x7f13e80b1ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:48:19.751 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:19.768+0000 7f13ef08d640 1 -- 172.21.15.88:0/2450248947 shutdown_connections 2024-03-29T10:48:19.751 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:19.768+0000 7f13ef08d640 1 -- 172.21.15.88:0/2450248947 wait complete. 2024-03-29T10:48:19.753 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:19.771+0000 7f13ef08d640 1 librados: shutdown 2024-03-29T10:48:19.772 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:48:26.579 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:48:26.735 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:26.753+0000 7f813ef72640 10 monclient: get_monmap_and_config 2024-03-29T10:48:26.736 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:26.753+0000 7f813ef72640 10 monclient: build_initial_monmap 2024-03-29T10:48:26.736 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:26.753+0000 7f813ef72640 10 monclient: monmap: 2024-03-29T10:48:26.736 INFO:teuthology.misc.health.smithi088.stderr:epoch 0 2024-03-29T10:48:26.736 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:48:26.736 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:48:26.754497+0000 2024-03-29T10:48:26.736 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:48:26.754497+0000 2024-03-29T10:48:26.736 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 0 (unknown) 2024-03-29T10:48:26.736 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:48:26.736 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-03-29T10:48:26.736 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-29T10:48:26.736 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-b 2024-03-29T10:48:26.737 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:48:26.744 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:26.754+0000 7f813ef72640 1 Processor -- start 2024-03-29T10:48:26.744 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:26.754+0000 7f813ef72640 1 -- start start 2024-03-29T10:48:26.744 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:26.754+0000 7f813ef72640 10 monclient: init 2024-03-29T10:48:26.744 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:26.755+0000 7f813ef72640 10 monclient: _reopen_session rank -1 2024-03-29T10:48:26.744 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:26.755+0000 7f813ef72640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:48:26.744 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:26.755+0000 7f813ef72640 10 monclient(hunting): picked mon.noname-b con 0x7f81381457f0 addr v1:172.21.15.88:6790/0 2024-03-29T10:48:26.744 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:26.755+0000 7f813ef72640 10 monclient(hunting): picked mon.noname-a con 0x7f813813f650 addr v1:172.21.15.88:6789/0 2024-03-29T10:48:26.744 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:26.755+0000 7f813ef72640 10 monclient(hunting): picked mon.noname-c con 0x7f81380a9d70 addr v1:172.21.15.196:6789/0 2024-03-29T10:48:26.744 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:26.755+0000 7f813ef72640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81380aaef0 con 0x7f813813f650 2024-03-29T10:48:26.744 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:26.755+0000 7f813ef72640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8138145bd0 con 0x7f81380a9d70 2024-03-29T10:48:26.744 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:26.755+0000 7f813ef72640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8138145eb0 con 0x7f81381457f0 2024-03-29T10:48:26.744 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:26.755+0000 7f813ef72640 10 monclient(hunting): _renew_subs 2024-03-29T10:48:26.745 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:26.755+0000 7f813ef72640 10 monclient(hunting): authenticate will time out at 1228.991534s 2024-03-29T10:48:26.745 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:26.755+0000 7f8137fff640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f813813f650 0x7f81380b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:53320/0 (socket says 172.21.15.88:53320) 2024-03-29T10:48:26.745 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:26.755+0000 7f8137fff640 1 -- 172.21.15.88:0/1338224641 learned_addr learned my addr 172.21.15.88:0/1338224641 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-29T10:48:26.745 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:26.756+0000 7f81377fe640 1 -- 172.21.15.88:0/1338224641 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 977496183 0 0) 0x7f8138145bd0 con 0x7f81380a9d70 2024-03-29T10:48:26.745 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:26.756+0000 7f81377fe640 10 monclient(hunting): _negotiate 2024-03-29T10:48:26.745 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:26.756+0000 7f81377fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:48:26.745 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:26.756+0000 7f81377fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:48:26.745 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:26.756+0000 7f81377fe640 10 monclient(hunting): my global_id is 4489 2024-03-29T10:48:26.745 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:26.756+0000 7f81377fe640 1 -- 172.21.15.88:0/1338224641 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8128003520 con 0x7f81380a9d70 2024-03-29T10:48:26.745 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:26.757+0000 7f81377fe640 1 -- 172.21.15.88:0/1338224641 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1670012907 0 0) 0x7f81380aaef0 con 0x7f813813f650 2024-03-29T10:48:26.745 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:26.757+0000 7f81377fe640 10 monclient(hunting): _negotiate 2024-03-29T10:48:26.745 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:26.757+0000 7f81377fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:48:26.745 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:26.757+0000 7f81377fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:48:26.746 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:26.757+0000 7f81377fe640 10 monclient(hunting): my global_id is 4503 2024-03-29T10:48:26.746 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:26.757+0000 7f81377fe640 1 -- 172.21.15.88:0/1338224641 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8138145bd0 con 0x7f813813f650 2024-03-29T10:48:26.746 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:26.757+0000 7f81377fe640 1 -- 172.21.15.88:0/1338224641 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1965228230 0 0) 0x7f8138145eb0 con 0x7f81381457f0 2024-03-29T10:48:26.746 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:26.757+0000 7f81377fe640 10 monclient(hunting): _negotiate 2024-03-29T10:48:26.746 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:26.757+0000 7f81377fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:48:26.746 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:26.757+0000 7f81377fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:48:26.746 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:26.757+0000 7f81377fe640 10 monclient(hunting): my global_id is 4535 2024-03-29T10:48:26.746 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:26.757+0000 7f81377fe640 1 -- 172.21.15.88:0/1338224641 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f81380aaef0 con 0x7f81381457f0 2024-03-29T10:48:26.746 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:26.757+0000 7f81377fe640 1 -- 172.21.15.88:0/1338224641 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3057223608 0 0) 0x7f8128003520 con 0x7f81380a9d70 2024-03-29T10:48:26.746 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:26.757+0000 7f81377fe640 1 -- 172.21.15.88:0/1338224641 >> v1:172.21.15.88:6790/0 conn(0x7f81381457f0 legacy=0x7f813813ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:26.746 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:26.757+0000 7f81377fe640 1 -- 172.21.15.88:0/1338224641 >> v1:172.21.15.88:6789/0 conn(0x7f813813f650 legacy=0x7f81380b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:26.746 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:26.757+0000 7f81377fe640 10 monclient: _finish_hunting 0 2024-03-29T10:48:26.746 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:26.757+0000 7f81377fe640 1 monclient: found mon.noname-c 2024-03-29T10:48:26.746 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:26.757+0000 7f81377fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:48:26.747 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:26.757+0000 7f81377fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.196:6789/0 2024-03-29T10:48:26.747 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:26.757+0000 7f81377fe640 1 -- 172.21.15.88:0/1338224641 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8138146340 con 0x7f81380a9d70 2024-03-29T10:48:26.747 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:26.757+0000 7f81377fe640 10 monclient: _finish_auth 0 2024-03-29T10:48:26.747 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:26.757+0000 7f81377fe640 10 monclient: _check_auth_tickets 2024-03-29T10:48:26.747 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:26.757+0000 7f81377fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:48:26.747 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:26.757+0000 7f81377fe640 1 -- 172.21.15.88:0/1338224641 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f8130001a10 con 0x7f81380a9d70 2024-03-29T10:48:26.747 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:26.757+0000 7f81377fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:26.747 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:26.757+0000 7f81377fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:48:26.747 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:26.757+0000 7f81377fe640 10 monclient: dump: 2024-03-29T10:48:26.747 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:48:26.747 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:48:26.747 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:48:26.747 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:48:26.747 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:26.748 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:48:26.748 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:48:26.748 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:48:26.748 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:48:26.748 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:48:26.748 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:26.757+0000 7f81377fe640 1 -- 172.21.15.88:0/1338224641 <== mon.1 v1:172.21.15.196:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8130001b70 con 0x7f81380a9d70 2024-03-29T10:48:26.748 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:26.757+0000 7f81377fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:48:26.748 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:26.757+0000 7f81377fe640 1 -- 172.21.15.88:0/1338224641 <== mon.1 v1:172.21.15.196:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f8130001f80 con 0x7f81380a9d70 2024-03-29T10:48:26.748 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:26.757+0000 7f81377fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:26.748 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:26.757+0000 7f81377fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:48:26.748 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:26.757+0000 7f81377fe640 10 monclient: dump: 2024-03-29T10:48:26.748 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:48:26.748 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:48:26.748 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:48:26.749 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:48:26.749 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:26.749 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:48:26.749 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:48:26.749 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:48:26.749 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:48:26.749 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:48:26.749 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:26.757+0000 7f813ef72640 5 monclient: authenticate success, global_id 4489 2024-03-29T10:48:26.749 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:26.757+0000 7f813ef72640 10 monclient: get_monmap_and_config success 2024-03-29T10:48:26.749 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:26.757+0000 7f813ef72640 4 set_mon_vals no callback set 2024-03-29T10:48:26.749 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:26.759+0000 7f813ef72640 10 monclient: shutdown 2024-03-29T10:48:26.749 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:26.759+0000 7f813ef72640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:48:26.749 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:26.759+0000 7f813ef72640 1 -- 172.21.15.88:0/1338224641 >> v1:172.21.15.196:6789/0 conn(0x7f81380a9d70 legacy=0x7f813814d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:26.749 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:26.759+0000 7f813ef72640 1 -- 172.21.15.88:0/1338224641 shutdown_connections 2024-03-29T10:48:26.749 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:26.759+0000 7f813ef72640 1 -- 172.21.15.88:0/1338224641 >> 172.21.15.88:0/1338224641 conn(0x7f8138066a00 msgr2=0x7f8138066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:48:26.750 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:26.760+0000 7f813ef72640 1 -- 172.21.15.88:0/1338224641 shutdown_connections 2024-03-29T10:48:26.750 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:26.760+0000 7f813ef72640 1 -- 172.21.15.88:0/1338224641 wait complete. 2024-03-29T10:48:26.750 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:26.761+0000 7f813ef72640 10 monclient: build_initial_monmap 2024-03-29T10:48:26.750 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:26.761+0000 7f813ef72640 10 monclient: monmap: 2024-03-29T10:48:26.750 INFO:teuthology.misc.health.smithi088.stderr:epoch 0 2024-03-29T10:48:26.750 INFO:teuthology.misc.health.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:48:26.750 INFO:teuthology.misc.health.smithi088.stderr:last_changed 0.000000 2024-03-29T10:48:26.750 INFO:teuthology.misc.health.smithi088.stderr:created 0.000000 2024-03-29T10:48:26.750 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 0 (unknown) 2024-03-29T10:48:26.750 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:48:26.750 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-03-29T10:48:26.751 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-29T10:48:26.751 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-c 2024-03-29T10:48:26.751 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:48:26.751 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:26.762+0000 7f813ef72640 1 librados: starting msgr at 2024-03-29T10:48:26.751 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:26.762+0000 7f813ef72640 1 librados: starting objecter 2024-03-29T10:48:26.756 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:26.762+0000 7f813ef72640 1 Processor -- start 2024-03-29T10:48:26.756 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:26.762+0000 7f813ef72640 1 -- start start 2024-03-29T10:48:26.756 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:26.762+0000 7f813ef72640 1 librados: setting wanted keys 2024-03-29T10:48:26.756 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:26.762+0000 7f813ef72640 1 librados: calling monclient init 2024-03-29T10:48:26.756 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:26.762+0000 7f813ef72640 10 monclient: init 2024-03-29T10:48:26.756 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:26.763+0000 7f813ef72640 10 monclient: _reopen_session rank -1 2024-03-29T10:48:26.756 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:26.763+0000 7f813ef72640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:48:26.756 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:26.763+0000 7f813ef72640 10 monclient(hunting): picked mon.noname-b con 0x7f81380a9d70 addr v1:172.21.15.196:6789/0 2024-03-29T10:48:26.756 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:26.763+0000 7f813ef72640 10 monclient(hunting): picked mon.noname-a con 0x7f813813f650 addr v1:172.21.15.88:6789/0 2024-03-29T10:48:26.756 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:26.763+0000 7f813ef72640 10 monclient(hunting): picked mon.noname-c con 0x7f81381457f0 addr v1:172.21.15.88:6790/0 2024-03-29T10:48:26.757 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:26.763+0000 7f813ef72640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81380ac030 con 0x7f813813f650 2024-03-29T10:48:26.757 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:26.763+0000 7f813ef72640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81380ad170 con 0x7f81380a9d70 2024-03-29T10:48:26.757 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:26.763+0000 7f813ef72640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81380ad450 con 0x7f81381457f0 2024-03-29T10:48:26.757 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:26.763+0000 7f813ef72640 10 monclient(hunting): _renew_subs 2024-03-29T10:48:26.757 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:26.763+0000 7f813ef72640 10 monclient(hunting): authenticate will time out at 1228.999874s 2024-03-29T10:48:26.757 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:26.764+0000 7f813cd0e640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f81380a9d70 0x7f81380b0bb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.88:51284/0 (socket says 172.21.15.88:51284) 2024-03-29T10:48:26.757 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:26.764+0000 7f813cd0e640 1 -- 172.21.15.88:0/3694700808 learned_addr learned my addr 172.21.15.88:0/3694700808 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-29T10:48:26.757 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:26.764+0000 7f81357fa640 10 client.?.objecter ms_handle_connect 0x7f81380a9d70 2024-03-29T10:48:26.757 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:26.764+0000 7f81357fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:48:26.757 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:26.764+0000 7f81357fa640 10 client.?.objecter ms_handle_connect 0x7f813813f650 2024-03-29T10:48:26.757 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:26.764+0000 7f81357fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:48:26.757 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:26.764+0000 7f81357fa640 1 -- 172.21.15.88:0/3694700808 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 256742331 0 0) 0x7f81380ad170 con 0x7f81380a9d70 2024-03-29T10:48:26.757 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:26.764+0000 7f81357fa640 10 monclient(hunting): _negotiate 2024-03-29T10:48:26.757 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:26.764+0000 7f81357fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:48:26.758 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:26.764+0000 7f81357fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:48:26.758 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:26.764+0000 7f81357fa640 10 monclient(hunting): my global_id is 4492 2024-03-29T10:48:26.758 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:26.764+0000 7f81357fa640 1 -- 172.21.15.88:0/3694700808 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f811c004eb0 con 0x7f81380a9d70 2024-03-29T10:48:26.758 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:26.764+0000 7f81357fa640 10 client.?.objecter ms_handle_connect 0x7f81381457f0 2024-03-29T10:48:26.758 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:26.764+0000 7f81357fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:48:26.758 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:26.764+0000 7f81357fa640 1 -- 172.21.15.88:0/3694700808 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3071105306 0 0) 0x7f811c004eb0 con 0x7f81380a9d70 2024-03-29T10:48:26.758 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:26.765+0000 7f81357fa640 1 -- 172.21.15.88:0/3694700808 --> v1:172.21.15.196:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f81380ad170 con 0x7f81380a9d70 2024-03-29T10:48:26.758 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:26.765+0000 7f81357fa640 1 -- 172.21.15.88:0/3694700808 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f812c001970 con 0x7f81380a9d70 2024-03-29T10:48:26.758 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:26.765+0000 7f81357fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:26.758 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:26.765+0000 7f81357fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:48:26.758 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:26.765+0000 7f81357fa640 10 monclient(hunting): dump: 2024-03-29T10:48:26.758 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:48:26.758 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:48:26.758 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:48:26.759 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:48:26.759 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:26.759 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:48:26.759 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:48:26.759 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:48:26.759 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:48:26.759 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:48:26.759 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:26.765+0000 7f81357fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:48:26.759 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:26.765+0000 7f81357fa640 1 -- 172.21.15.88:0/3694700808 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 224795036 0 0) 0x7f81380ad450 con 0x7f81381457f0 2024-03-29T10:48:26.759 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:26.765+0000 7f81357fa640 10 monclient(hunting): _negotiate 2024-03-29T10:48:26.759 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:26.765+0000 7f81357fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:48:26.759 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:26.765+0000 7f81357fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:48:26.759 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:26.765+0000 7f81357fa640 10 monclient(hunting): my global_id is 4538 2024-03-29T10:48:26.759 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:26.765+0000 7f81357fa640 1 -- 172.21.15.88:0/3694700808 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f811c004eb0 con 0x7f81381457f0 2024-03-29T10:48:26.760 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:26.765+0000 7f81357fa640 1 -- 172.21.15.88:0/3694700808 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 185170934 0 0) 0x7f81380ac030 con 0x7f813813f650 2024-03-29T10:48:26.760 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:26.765+0000 7f81357fa640 10 monclient(hunting): _negotiate 2024-03-29T10:48:26.760 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:26.765+0000 7f81357fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:48:26.760 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:26.765+0000 7f81357fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:48:26.760 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:26.765+0000 7f81357fa640 10 monclient(hunting): my global_id is 4506 2024-03-29T10:48:26.760 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:26.765+0000 7f81357fa640 1 -- 172.21.15.88:0/3694700808 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f81380ad450 con 0x7f813813f650 2024-03-29T10:48:26.760 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:26.765+0000 7f81357fa640 1 -- 172.21.15.88:0/3694700808 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 958622312 0 0) 0x7f811c004eb0 con 0x7f81381457f0 2024-03-29T10:48:26.760 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:26.765+0000 7f81357fa640 1 -- 172.21.15.88:0/3694700808 --> v1:172.21.15.88:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f81380ac030 con 0x7f81381457f0 2024-03-29T10:48:26.760 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:26.765+0000 7f81357fa640 1 -- 172.21.15.88:0/3694700808 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f8130001ce0 con 0x7f81381457f0 2024-03-29T10:48:26.760 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:26.765+0000 7f81357fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:26.760 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:26.765+0000 7f81357fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:48:26.760 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:26.765+0000 7f81357fa640 10 monclient(hunting): dump: 2024-03-29T10:48:26.760 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:48:26.760 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:48:26.761 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:48:26.761 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:48:26.761 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:26.761 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:48:26.761 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:48:26.761 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:48:26.761 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:48:26.761 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:48:26.761 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:26.765+0000 7f81357fa640 1 -- 172.21.15.88:0/3694700808 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 89274609 0 0) 0x7f81380ad450 con 0x7f813813f650 2024-03-29T10:48:26.761 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:26.766+0000 7f81357fa640 1 -- 172.21.15.88:0/3694700808 --> v1:172.21.15.88:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f811c004eb0 con 0x7f813813f650 2024-03-29T10:48:26.761 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:26.766+0000 7f81357fa640 1 -- 172.21.15.88:0/3694700808 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f8124001930 con 0x7f813813f650 2024-03-29T10:48:26.761 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:26.766+0000 7f81357fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:26.761 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:26.766+0000 7f81357fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:48:26.761 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:26.766+0000 7f81357fa640 10 monclient(hunting): dump: 2024-03-29T10:48:26.761 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:48:26.762 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:48:26.762 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:48:26.762 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:48:26.762 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:26.762 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:48:26.762 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:48:26.762 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:48:26.762 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:48:26.762 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:48:26.762 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:26.766+0000 7f81357fa640 1 -- 172.21.15.88:0/3694700808 <== mon.1 v1:172.21.15.196:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 507063272 0 0) 0x7f81380ad170 con 0x7f81380a9d70 2024-03-29T10:48:26.762 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:26.766+0000 7f81357fa640 1 -- 172.21.15.88:0/3694700808 >> v1:172.21.15.88:6790/0 conn(0x7f81381457f0 legacy=0x7f81381d7a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:26.762 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:26.766+0000 7f81357fa640 1 -- 172.21.15.88:0/3694700808 >> v1:172.21.15.88:6789/0 conn(0x7f813813f650 legacy=0x7f81381d72f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:26.762 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:26.766+0000 7f81357fa640 10 monclient: _finish_hunting 0 2024-03-29T10:48:26.762 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:26.766+0000 7f81357fa640 1 monclient: found mon.b 2024-03-29T10:48:26.763 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:26.766+0000 7f81357fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:48:26.763 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:26.766+0000 7f81357fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T10:48:26.763 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:26.766+0000 7f81357fa640 1 -- 172.21.15.88:0/3694700808 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81381d4d30 con 0x7f81380a9d70 2024-03-29T10:48:26.763 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:26.766+0000 7f81357fa640 10 monclient: _finish_auth 0 2024-03-29T10:48:26.763 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:26.766+0000 7f81357fa640 10 monclient: _check_auth_tickets 2024-03-29T10:48:26.763 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:26.766+0000 7f81357fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:48:26.763 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:26.766+0000 7f813ef72640 5 monclient: authenticate success, global_id 4492 2024-03-29T10:48:26.763 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:26.766+0000 7f813ef72640 10 monclient: _renew_subs 2024-03-29T10:48:26.763 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:26.766+0000 7f813ef72640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T10:48:26.763 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:26.766+0000 7f813ef72640 1 -- 172.21.15.88:0/3694700808 --> v1:172.21.15.196:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81380abae0 con 0x7f81380a9d70 2024-03-29T10:48:26.763 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:26.766+0000 7f813ef72640 10 client.4492.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:48:26.763 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:26.766+0000 7f813ef72640 10 monclient: _renew_subs 2024-03-29T10:48:26.763 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:26.766+0000 7f813ef72640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T10:48:26.763 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:26.766+0000 7f813ef72640 1 -- 172.21.15.88:0/3694700808 --> v1:172.21.15.196:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f813813bab0 con 0x7f81380a9d70 2024-03-29T10:48:26.764 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:26.766+0000 7f813ef72640 1 librados: init done 2024-03-29T10:48:26.764 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:26.767+0000 7f81357fa640 1 -- 172.21.15.88:0/3694700808 <== mon.1 v1:172.21.15.196:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f812c003610 con 0x7f81380a9d70 2024-03-29T10:48:26.764 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:26.767+0000 7f81357fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:48:26.764 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:26.767+0000 7f81367fc640 4 set_mon_vals no callback set 2024-03-29T10:48:26.764 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:26.767+0000 7f81357fa640 1 -- 172.21.15.88:0/3694700808 <== mon.1 v1:172.21.15.196:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f812c003a60 con 0x7f81380a9d70 2024-03-29T10:48:26.764 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:26.767+0000 7f81357fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:26.764 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:26.767+0000 7f81357fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:48:26.764 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:26.767+0000 7f81357fa640 10 monclient: dump: 2024-03-29T10:48:26.764 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:48:26.764 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:48:26.764 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:48:26.764 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:48:26.764 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:26.764 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:48:26.765 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:48:26.765 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:48:26.765 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:48:26.765 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:48:26.765 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:26.768+0000 7f813ef72640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:48:26.765 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:26.768+0000 7f813ef72640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:48:26.765 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:26.768+0000 7f813ef72640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T10:48:26.765 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:26.768+0000 7f813ef72640 1 -- 172.21.15.88:0/3694700808 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8118005080 con 0x7f81380a9d70 2024-03-29T10:48:26.765 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:26.769+0000 7f81357fa640 1 -- 172.21.15.88:0/3694700808 <== mon.1 v1:172.21.15.196:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4235629220 0 0) 0x7f812c003c40 con 0x7f81380a9d70 2024-03-29T10:48:26.765 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:26.770+0000 7f8137fff640 10 monclient: get_auth_request con 0x7f811c06e100 auth_method 0 2024-03-29T10:48:26.766 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:26.770+0000 7f81357fa640 1 -- 172.21.15.88:0/3694700808 <== mon.1 v1:172.21.15.196:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4257673582 0 0) 0x7f812c084b00 con 0x7f81380a9d70 2024-03-29T10:48:26.766 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:26.770+0000 7f81357fa640 10 client.4492.objecter ms_dispatch 0x7f8138000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:48:26.766 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:26.770+0000 7f81357fa640 3 client.4492.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:48:26.766 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:26.770+0000 7f81357fa640 3 client.4492.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:48:26.766 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:26.770+0000 7f81357fa640 20 client.4492.objecter dump_active .. 0 homeless 2024-03-29T10:48:26.766 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:26.770+0000 7f81357fa640 10 client.4492.objecter ms_handle_connect 0x7f811c06e100 2024-03-29T10:48:26.766 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:26.778+0000 7f81357fa640 1 -- 172.21.15.88:0/3694700808 <== mon.1 v1:172.21.15.196:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f812c0503e0 con 0x7f81380a9d70 2024-03-29T10:48:26.767 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:26.778+0000 7f81357fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:48:26.767 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:26.778+0000 7f81357fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:48:27.129 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:27.147+0000 7f813ef72640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:48:27.129 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:27.147+0000 7f813ef72640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:48:27.129 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:27.147+0000 7f813ef72640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T10:48:27.129 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:27.147+0000 7f813ef72640 1 -- 172.21.15.88:0/3694700808 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8118005330 con 0x7f81380a9d70 2024-03-29T10:48:27.131 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:27.149+0000 7f81357fa640 1 -- 172.21.15.88:0/3694700808 <== mon.1 v1:172.21.15.196:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 344506374) 0x7f812c052310 con 0x7f81380a9d70 2024-03-29T10:48:27.131 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:27.149+0000 7f81357fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:48:27.132 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:27.149+0000 7f81357fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:48:27.132 INFO:teuthology.misc.health.smithi088.stdout: 2024-03-29T10:48:27.132 INFO:teuthology.misc.health.smithi088.stdout:{"epoch":10,"fsid":"869f5188-f92b-42dd-a46d-8c646e0388a9","created":"2024-03-29T10:40:40.285087+0000","modified":"2024-03-29T10:40:42.536230+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:40:42.536230+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a26c0c1c-8de0-438c-a3f6-3df10ea61574","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5de24915-8dec-49b6-8b16-f3170197c591","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ac054af3-d589-4565-9644-e6af75975c53","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3ec4f045-c714-4a22-bd1a-e07330b45ab3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"cddb9867-8e30-4951-9b13-3d1e78b9c45a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dc6d9911-785d-432f-8b53-56a1ccab302f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"77c88437-b587-496e-8b4d-1824ea8b5305","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ce963d96-5a8e-44c3-99de-c0dbebe234ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:48:27.134 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:27.153+0000 7f813ef72640 10 librados: watch_flush enter 2024-03-29T10:48:27.135 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:27.153+0000 7f813ef72640 10 librados: watch_flush exit 2024-03-29T10:48:27.135 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:27.153+0000 7f813ef72640 20 client.4492.objecter shutdown clearing up homeless session... 2024-03-29T10:48:27.135 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:27.153+0000 7f813ef72640 10 client.4492.objecter successfully canceled tick 2024-03-29T10:48:27.135 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:27.153+0000 7f813ef72640 1 -- 172.21.15.88:0/3694700808 >> v1:172.21.15.88:6816/1752861349 conn(0x7f811c06e100 legacy=0x7f811c0705c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:27.135 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:27.153+0000 7f813ef72640 10 monclient: shutdown 2024-03-29T10:48:27.135 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:27.153+0000 7f813ef72640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:48:27.136 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:27.153+0000 7f813ef72640 1 -- 172.21.15.88:0/3694700808 >> v1:172.21.15.196:6789/0 conn(0x7f81380a9d70 legacy=0x7f81380b0bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:27.136 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:27.153+0000 7f813ef72640 1 -- 172.21.15.88:0/3694700808 shutdown_connections 2024-03-29T10:48:27.136 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:27.154+0000 7f813ef72640 1 -- 172.21.15.88:0/3694700808 >> 172.21.15.88:0/3694700808 conn(0x7f8138066a00 msgr2=0x7f813813cad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:48:27.136 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:27.154+0000 7f813ef72640 1 -- 172.21.15.88:0/3694700808 shutdown_connections 2024-03-29T10:48:27.136 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:27.154+0000 7f813ef72640 1 -- 172.21.15.88:0/3694700808 wait complete. 2024-03-29T10:48:27.137 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:27.154+0000 7f813ef72640 1 librados: shutdown 2024-03-29T10:48:27.151 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:48:33.958 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:48:34.105 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:34.113+0000 7f2f6177a640 10 monclient: get_monmap_and_config 2024-03-29T10:48:34.106 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:34.113+0000 7f2f6177a640 10 monclient: build_initial_monmap 2024-03-29T10:48:34.106 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:34.114+0000 7f2f6177a640 10 monclient: monmap: 2024-03-29T10:48:34.106 INFO:teuthology.misc.health.smithi088.stderr:epoch 0 2024-03-29T10:48:34.106 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:48:34.106 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:48:34.115024+0000 2024-03-29T10:48:34.106 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:48:34.115024+0000 2024-03-29T10:48:34.106 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 0 (unknown) 2024-03-29T10:48:34.106 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:48:34.106 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-03-29T10:48:34.106 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-29T10:48:34.107 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-b 2024-03-29T10:48:34.107 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:48:34.107 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:34.114+0000 7f2f6177a640 1 Processor -- start 2024-03-29T10:48:34.107 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:34.114+0000 7f2f6177a640 1 -- start start 2024-03-29T10:48:34.107 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:34.114+0000 7f2f6177a640 10 monclient: init 2024-03-29T10:48:34.107 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:34.115+0000 7f2f6177a640 10 monclient: _reopen_session rank -1 2024-03-29T10:48:34.107 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:34.115+0000 7f2f6177a640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:48:34.107 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:34.115+0000 7f2f6177a640 10 monclient(hunting): picked mon.noname-c con 0x7f2f5c0b1f60 addr v1:172.21.15.196:6789/0 2024-03-29T10:48:34.107 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:34.116+0000 7f2f6177a640 10 monclient(hunting): picked mon.noname-a con 0x7f2f5c071eb0 addr v1:172.21.15.88:6789/0 2024-03-29T10:48:34.107 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:34.116+0000 7f2f6177a640 10 monclient(hunting): picked mon.noname-b con 0x7f2f5c143d10 addr v1:172.21.15.88:6790/0 2024-03-29T10:48:34.107 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:34.116+0000 7f2f5b7fe640 1 --1- >> v1:172.21.15.88:6790/0 conn(0x7f2f5c143d10 0x7f2f5c146b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6790/0 says I am v1:172.21.15.88:60464/0 (socket says 172.21.15.88:60464) 2024-03-29T10:48:34.107 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:34.116+0000 7f2f5b7fe640 1 -- 172.21.15.88:0/1590761207 learned_addr learned my addr 172.21.15.88:0/1590761207 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-29T10:48:34.107 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:34.116+0000 7f2f6177a640 1 -- 172.21.15.88:0/1590761207 --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f5c06ef80 con 0x7f2f5c071eb0 2024-03-29T10:48:34.107 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:34.116+0000 7f2f6177a640 1 -- 172.21.15.88:0/1590761207 --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f5c06f260 con 0x7f2f5c0b1f60 2024-03-29T10:48:34.108 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:34.116+0000 7f2f6177a640 1 -- 172.21.15.88:0/1590761207 --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f5c0b40c0 con 0x7f2f5c143d10 2024-03-29T10:48:34.108 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:34.116+0000 7f2f6177a640 10 monclient(hunting): _renew_subs 2024-03-29T10:48:34.108 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:34.116+0000 7f2f6177a640 10 monclient(hunting): authenticate will time out at 1236.352664s 2024-03-29T10:48:34.108 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:34.116+0000 7f2f59ffb640 1 -- 172.21.15.88:0/1590761207 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1994245407 0 0) 0x7f2f5c0b40c0 con 0x7f2f5c143d10 2024-03-29T10:48:34.108 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:34.116+0000 7f2f59ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:48:34.108 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:34.116+0000 7f2f59ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:48:34.108 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:34.116+0000 7f2f59ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:48:34.108 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:34.116+0000 7f2f59ffb640 10 monclient(hunting): my global_id is 4541 2024-03-29T10:48:34.108 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:34.116+0000 7f2f59ffb640 1 -- 172.21.15.88:0/1590761207 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2f44003520 con 0x7f2f5c143d10 2024-03-29T10:48:34.108 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:34.117+0000 7f2f59ffb640 1 -- 172.21.15.88:0/1590761207 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1463372888 0 0) 0x7f2f44003520 con 0x7f2f5c143d10 2024-03-29T10:48:34.108 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:34.117+0000 7f2f59ffb640 1 -- 172.21.15.88:0/1590761207 >> v1:172.21.15.196:6789/0 conn(0x7f2f5c0b1f60 legacy=0x7f2f5c143600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:48:34.108 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:34.117+0000 7f2f59ffb640 1 -- 172.21.15.88:0/1590761207 >> v1:172.21.15.88:6789/0 conn(0x7f2f5c071eb0 legacy=0x7f2f5c146470 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:48:34.108 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:34.117+0000 7f2f59ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:48:34.108 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:34.117+0000 7f2f59ffb640 1 monclient: found mon.noname-b 2024-03-29T10:48:34.108 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:34.117+0000 7f2f59ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:48:34.109 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:34.117+0000 7f2f59ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.88:6790/0 2024-03-29T10:48:34.109 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:34.117+0000 7f2f59ffb640 1 -- 172.21.15.88:0/1590761207 --> v1:172.21.15.88:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f5c154d60 con 0x7f2f5c143d10 2024-03-29T10:48:34.109 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:34.117+0000 7f2f59ffb640 10 monclient: _finish_auth 0 2024-03-29T10:48:34.109 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:34.117+0000 7f2f59ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:48:34.109 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:34.117+0000 7f2f59ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:48:34.109 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:34.117+0000 7f2f59ffb640 1 -- 172.21.15.88:0/1590761207 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f2f54001a10 con 0x7f2f5c143d10 2024-03-29T10:48:34.109 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:34.118+0000 7f2f59ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:34.109 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:34.118+0000 7f2f59ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:48:34.109 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:34.118+0000 7f2f59ffb640 10 monclient: dump: 2024-03-29T10:48:34.109 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:48:34.109 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:48:34.109 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:48:34.109 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:48:34.109 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:34.110 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:48:34.110 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:48:34.110 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:48:34.110 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:48:34.110 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:48:34.110 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:34.118+0000 7f2f6177a640 5 monclient: authenticate success, global_id 4541 2024-03-29T10:48:34.110 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:34.118+0000 7f2f6177a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:48:34.110 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:34.118+0000 7f2f59ffb640 1 -- 172.21.15.88:0/1590761207 <== mon.2 v1:172.21.15.88:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2f54001c70 con 0x7f2f5c143d10 2024-03-29T10:48:34.110 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:34.118+0000 7f2f59ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:48:34.110 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:34.118+0000 7f2f59ffb640 1 -- 172.21.15.88:0/1590761207 <== mon.2 v1:172.21.15.88:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f2f54002080 con 0x7f2f5c143d10 2024-03-29T10:48:34.110 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:34.118+0000 7f2f6177a640 10 monclient: get_monmap_and_config success 2024-03-29T10:48:34.110 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:34.118+0000 7f2f6177a640 4 set_mon_vals no callback set 2024-03-29T10:48:34.110 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:34.118+0000 7f2f59ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:34.110 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:34.119+0000 7f2f59ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:48:34.111 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:34.119+0000 7f2f59ffb640 10 monclient: dump: 2024-03-29T10:48:34.111 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:48:34.111 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:48:34.111 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:48:34.111 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:48:34.111 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:34.111 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:48:34.111 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:48:34.111 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:48:34.111 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:48:34.111 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:48:34.111 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:34.119+0000 7f2f6177a640 10 monclient: shutdown 2024-03-29T10:48:34.111 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:34.119+0000 7f2f6177a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:48:34.111 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:34.119+0000 7f2f6177a640 1 -- 172.21.15.88:0/1590761207 >> v1:172.21.15.88:6790/0 conn(0x7f2f5c143d10 legacy=0x7f2f5c146b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:34.111 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:34.120+0000 7f2f6177a640 1 -- 172.21.15.88:0/1590761207 shutdown_connections 2024-03-29T10:48:34.111 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:34.120+0000 7f2f6177a640 1 -- 172.21.15.88:0/1590761207 >> 172.21.15.88:0/1590761207 conn(0x7f2f5c066a00 msgr2=0x7f2f5c066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:48:34.112 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:34.120+0000 7f2f6177a640 1 -- 172.21.15.88:0/1590761207 shutdown_connections 2024-03-29T10:48:34.112 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:34.120+0000 7f2f6177a640 1 -- 172.21.15.88:0/1590761207 wait complete. 2024-03-29T10:48:34.112 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:34.120+0000 7f2f6177a640 10 monclient: build_initial_monmap 2024-03-29T10:48:34.112 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:34.120+0000 7f2f6177a640 10 monclient: monmap: 2024-03-29T10:48:34.112 INFO:teuthology.misc.health.smithi088.stderr:epoch 0 2024-03-29T10:48:34.112 INFO:teuthology.misc.health.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:48:34.112 INFO:teuthology.misc.health.smithi088.stderr:last_changed 0.000000 2024-03-29T10:48:34.112 INFO:teuthology.misc.health.smithi088.stderr:created 0.000000 2024-03-29T10:48:34.112 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 0 (unknown) 2024-03-29T10:48:34.112 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:48:34.112 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-03-29T10:48:34.112 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-29T10:48:34.112 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-c 2024-03-29T10:48:34.112 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:48:34.113 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:34.121+0000 7f2f6177a640 1 librados: starting msgr at 2024-03-29T10:48:34.113 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:34.121+0000 7f2f6177a640 1 librados: starting objecter 2024-03-29T10:48:34.113 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:34.121+0000 7f2f6177a640 1 Processor -- start 2024-03-29T10:48:34.113 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:34.121+0000 7f2f6177a640 1 -- start start 2024-03-29T10:48:34.113 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:34.121+0000 7f2f6177a640 1 librados: setting wanted keys 2024-03-29T10:48:34.113 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:34.121+0000 7f2f6177a640 1 librados: calling monclient init 2024-03-29T10:48:34.113 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:34.121+0000 7f2f6177a640 10 monclient: init 2024-03-29T10:48:34.113 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:34.122+0000 7f2f6177a640 10 monclient: _reopen_session rank -1 2024-03-29T10:48:34.113 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:34.122+0000 7f2f6177a640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:48:34.113 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:34.122+0000 7f2f6177a640 10 monclient(hunting): picked mon.noname-c con 0x7f2f5c0b1f60 addr v1:172.21.15.88:6790/0 2024-03-29T10:48:34.113 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:34.122+0000 7f2f6177a640 10 monclient(hunting): picked mon.noname-a con 0x7f2f5c143d10 addr v1:172.21.15.88:6789/0 2024-03-29T10:48:34.113 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:34.122+0000 7f2f6177a640 10 monclient(hunting): picked mon.noname-b con 0x7f2f5c0b5ac0 addr v1:172.21.15.196:6789/0 2024-03-29T10:48:34.113 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:34.123+0000 7f2f6177a640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f5c06fac0 con 0x7f2f5c143d10 2024-03-29T10:48:34.113 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:34.123+0000 7f2f6177a640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f5c0b6ee0 con 0x7f2f5c0b5ac0 2024-03-29T10:48:34.113 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:34.123+0000 7f2f6177a640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f5c0717f0 con 0x7f2f5c0b1f60 2024-03-29T10:48:34.114 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:34.123+0000 7f2f6177a640 10 monclient(hunting): _renew_subs 2024-03-29T10:48:34.114 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:34.123+0000 7f2f6177a640 10 monclient(hunting): authenticate will time out at 1236.358940s 2024-03-29T10:48:34.114 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:34.123+0000 7f2f5b7fe640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f2f5c0b5ac0 0x7f2f5c1e6120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.88:38240/0 (socket says 172.21.15.88:38240) 2024-03-29T10:48:34.114 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:34.123+0000 7f2f5b7fe640 1 -- 172.21.15.88:0/3377064795 learned_addr learned my addr 172.21.15.88:0/3377064795 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-29T10:48:34.114 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:34.123+0000 7f2f437fe640 10 client.?.objecter ms_handle_connect 0x7f2f5c0b5ac0 2024-03-29T10:48:34.114 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:34.123+0000 7f2f437fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:48:34.114 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:34.123+0000 7f2f437fe640 10 client.?.objecter ms_handle_connect 0x7f2f5c0b1f60 2024-03-29T10:48:34.114 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:34.123+0000 7f2f437fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:48:34.114 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:34.123+0000 7f2f437fe640 1 -- 172.21.15.88:0/3377064795 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2817125562 0 0) 0x7f2f5c0b6ee0 con 0x7f2f5c0b5ac0 2024-03-29T10:48:34.114 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:34.123+0000 7f2f437fe640 10 monclient(hunting): _negotiate 2024-03-29T10:48:34.114 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:34.123+0000 7f2f437fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:48:34.114 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:34.123+0000 7f2f437fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:48:34.114 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:34.123+0000 7f2f437fe640 10 monclient(hunting): my global_id is 4495 2024-03-29T10:48:34.114 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:34.123+0000 7f2f437fe640 1 -- 172.21.15.88:0/3377064795 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2f48004eb0 con 0x7f2f5c0b5ac0 2024-03-29T10:48:34.120 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:34.123+0000 7f2f437fe640 1 -- 172.21.15.88:0/3377064795 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 289744289 0 0) 0x7f2f5c0717f0 con 0x7f2f5c0b1f60 2024-03-29T10:48:34.121 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:34.124+0000 7f2f437fe640 10 monclient(hunting): _negotiate 2024-03-29T10:48:34.121 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:34.124+0000 7f2f437fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:48:34.121 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:34.124+0000 7f2f437fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:48:34.121 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:34.124+0000 7f2f437fe640 10 monclient(hunting): my global_id is 4544 2024-03-29T10:48:34.121 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:34.124+0000 7f2f437fe640 1 -- 172.21.15.88:0/3377064795 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2f5c0b6ee0 con 0x7f2f5c0b1f60 2024-03-29T10:48:34.121 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:34.124+0000 7f2f437fe640 1 -- 172.21.15.88:0/3377064795 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3660505820 0 0) 0x7f2f48004eb0 con 0x7f2f5c0b5ac0 2024-03-29T10:48:34.121 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:34.124+0000 7f2f437fe640 1 -- 172.21.15.88:0/3377064795 --> v1:172.21.15.196:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2f5c0717f0 con 0x7f2f5c0b5ac0 2024-03-29T10:48:34.121 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:34.125+0000 7f2f437fe640 1 -- 172.21.15.88:0/3377064795 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f2f540019d0 con 0x7f2f5c0b5ac0 2024-03-29T10:48:34.121 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:34.125+0000 7f2f437fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:34.121 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:34.125+0000 7f2f437fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:48:34.121 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:34.125+0000 7f2f437fe640 10 monclient(hunting): dump: 2024-03-29T10:48:34.121 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:48:34.122 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:48:34.122 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:48:34.122 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:48:34.122 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:34.122 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:48:34.122 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:48:34.122 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:48:34.122 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:48:34.122 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:48:34.122 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:34.125+0000 7f2f437fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:48:34.122 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:34.125+0000 7f2f437fe640 1 -- 172.21.15.88:0/3377064795 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 406202983 0 0) 0x7f2f5c0b6ee0 con 0x7f2f5c0b1f60 2024-03-29T10:48:34.122 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:34.125+0000 7f2f437fe640 1 -- 172.21.15.88:0/3377064795 --> v1:172.21.15.88:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2f48004eb0 con 0x7f2f5c0b1f60 2024-03-29T10:48:34.122 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:34.125+0000 7f2f437fe640 10 client.?.objecter ms_handle_connect 0x7f2f5c143d10 2024-03-29T10:48:34.122 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:34.125+0000 7f2f437fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:48:34.123 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:34.125+0000 7f2f437fe640 1 -- 172.21.15.88:0/3377064795 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f2f4c0017b0 con 0x7f2f5c0b1f60 2024-03-29T10:48:34.123 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:34.125+0000 7f2f437fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:34.123 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:34.125+0000 7f2f437fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:48:34.123 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:34.125+0000 7f2f437fe640 10 monclient(hunting): dump: 2024-03-29T10:48:34.123 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:48:34.123 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:48:34.123 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:48:34.123 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:48:34.123 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:34.123 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:48:34.123 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:48:34.123 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:48:34.123 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:48:34.123 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:48:34.124 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:34.125+0000 7f2f437fe640 1 -- 172.21.15.88:0/3377064795 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2928483451 0 0) 0x7f2f5c06fac0 con 0x7f2f5c143d10 2024-03-29T10:48:34.124 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:34.125+0000 7f2f437fe640 10 monclient(hunting): _negotiate 2024-03-29T10:48:34.124 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:34.125+0000 7f2f437fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:48:34.124 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:34.125+0000 7f2f437fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:48:34.124 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:34.125+0000 7f2f437fe640 10 monclient(hunting): my global_id is 4512 2024-03-29T10:48:34.124 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:34.125+0000 7f2f437fe640 1 -- 172.21.15.88:0/3377064795 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2f5c0b6ee0 con 0x7f2f5c143d10 2024-03-29T10:48:34.124 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:34.125+0000 7f2f437fe640 1 -- 172.21.15.88:0/3377064795 <== mon.2 v1:172.21.15.88:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 113172237 0 0) 0x7f2f48004eb0 con 0x7f2f5c0b1f60 2024-03-29T10:48:34.124 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:34.125+0000 7f2f437fe640 1 -- 172.21.15.88:0/3377064795 >> v1:172.21.15.196:6789/0 conn(0x7f2f5c0b5ac0 legacy=0x7f2f5c1e6120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:34.124 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:34.125+0000 7f2f437fe640 1 -- 172.21.15.88:0/3377064795 >> v1:172.21.15.88:6789/0 conn(0x7f2f5c143d10 legacy=0x7f2f5c144270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:34.124 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:34.126+0000 7f2f437fe640 10 monclient: _finish_hunting 0 2024-03-29T10:48:34.124 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:34.126+0000 7f2f437fe640 1 monclient: found mon.c 2024-03-29T10:48:34.124 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:34.126+0000 7f2f437fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:48:34.124 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:34.126+0000 7f2f437fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-03-29T10:48:34.124 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:34.126+0000 7f2f437fe640 1 -- 172.21.15.88:0/3377064795 --> v1:172.21.15.88:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f5c0711d0 con 0x7f2f5c0b1f60 2024-03-29T10:48:34.125 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:34.126+0000 7f2f437fe640 10 monclient: _finish_auth 0 2024-03-29T10:48:34.125 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:34.126+0000 7f2f437fe640 10 monclient: _check_auth_tickets 2024-03-29T10:48:34.125 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:34.126+0000 7f2f437fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:48:34.125 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:34.126+0000 7f2f437fe640 1 -- 172.21.15.88:0/3377064795 <== mon.2 v1:172.21.15.88:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2f4c001910 con 0x7f2f5c0b1f60 2024-03-29T10:48:34.125 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:34.126+0000 7f2f437fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:48:34.125 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:34.126+0000 7f2f437fe640 1 -- 172.21.15.88:0/3377064795 <== mon.2 v1:172.21.15.88:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f2f4c003790 con 0x7f2f5c0b1f60 2024-03-29T10:48:34.125 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:34.126+0000 7f2f437fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:34.125 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:34.126+0000 7f2f437fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:48:34.125 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:34.126+0000 7f2f437fe640 10 monclient: dump: 2024-03-29T10:48:34.125 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:48:34.125 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:48:34.125 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:48:34.125 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:48:34.125 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:34.125 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:48:34.126 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:48:34.126 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:48:34.126 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:48:34.126 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:48:34.126 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:34.127+0000 7f2f6177a640 5 monclient: authenticate success, global_id 4544 2024-03-29T10:48:34.126 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:34.127+0000 7f2f6177a640 10 monclient: _renew_subs 2024-03-29T10:48:34.126 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:34.127+0000 7f2f6177a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-03-29T10:48:34.126 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:34.127+0000 7f2f6177a640 1 -- 172.21.15.88:0/3377064795 --> v1:172.21.15.88:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f5c144980 con 0x7f2f5c0b1f60 2024-03-29T10:48:34.126 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:34.127+0000 7f2f6177a640 10 client.4544.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:48:34.126 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:34.127+0000 7f2f6177a640 10 monclient: _renew_subs 2024-03-29T10:48:34.126 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:34.127+0000 7f2f6177a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-03-29T10:48:34.126 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:34.127+0000 7f2f6177a640 1 -- 172.21.15.88:0/3377064795 --> v1:172.21.15.88:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f5c144d20 con 0x7f2f5c0b1f60 2024-03-29T10:48:34.127 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:34.127+0000 7f2f6177a640 1 librados: init done 2024-03-29T10:48:34.127 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:34.128+0000 7f2f597fa640 4 set_mon_vals no callback set 2024-03-29T10:48:34.127 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:34.129+0000 7f2f437fe640 1 -- 172.21.15.88:0/3377064795 <== mon.2 v1:172.21.15.88:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4235629220 0 0) 0x7f2f4c019760 con 0x7f2f5c0b1f60 2024-03-29T10:48:34.127 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:34.129+0000 7f2f437fe640 1 -- 172.21.15.88:0/3377064795 <== mon.2 v1:172.21.15.88:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4257673582 0 0) 0x7f2f4c0836e0 con 0x7f2f5c0b1f60 2024-03-29T10:48:34.127 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:34.129+0000 7f2f437fe640 10 client.4544.objecter ms_dispatch 0x7f2f5c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:48:34.127 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:34.129+0000 7f2f437fe640 3 client.4544.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:48:34.127 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:34.129+0000 7f2f437fe640 3 client.4544.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:48:34.127 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:34.129+0000 7f2f437fe640 20 client.4544.objecter dump_active .. 0 homeless 2024-03-29T10:48:34.127 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:34.130+0000 7f2f5a7fc640 10 monclient: get_auth_request con 0x7f2f4806dfe0 auth_method 0 2024-03-29T10:48:34.127 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:34.130+0000 7f2f437fe640 10 client.4544.objecter ms_handle_connect 0x7f2f4806dfe0 2024-03-29T10:48:34.127 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:34.130+0000 7f2f6177a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:48:34.127 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:34.130+0000 7f2f6177a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:48:34.127 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:34.130+0000 7f2f6177a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-03-29T10:48:34.127 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:34.130+0000 7f2f6177a640 1 -- 172.21.15.88:0/3377064795 --> v1:172.21.15.88:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f3c005080 con 0x7f2f5c0b1f60 2024-03-29T10:48:34.128 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:34.137+0000 7f2f437fe640 1 -- 172.21.15.88:0/3377064795 <== mon.2 v1:172.21.15.88:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2f4c04efc0 con 0x7f2f5c0b1f60 2024-03-29T10:48:34.128 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:34.137+0000 7f2f437fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:48:34.128 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:34.137+0000 7f2f437fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:48:34.422 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:34.439+0000 7f2f6177a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:48:34.422 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:34.440+0000 7f2f6177a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:48:34.422 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:34.440+0000 7f2f6177a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-03-29T10:48:34.422 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:34.440+0000 7f2f6177a640 1 -- 172.21.15.88:0/3377064795 --> v1:172.21.15.88:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2f3c005330 con 0x7f2f5c0b1f60 2024-03-29T10:48:34.422 INFO:teuthology.misc.health.smithi088.stdout: 2024-03-29T10:48:34.423 INFO:teuthology.misc.health.smithi088.stdout:{"epoch":10,"fsid":"869f5188-f92b-42dd-a46d-8c646e0388a9","created":"2024-03-29T10:40:40.285087+0000","modified":"2024-03-29T10:40:42.536230+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:40:42.536230+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a26c0c1c-8de0-438c-a3f6-3df10ea61574","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5de24915-8dec-49b6-8b16-f3170197c591","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ac054af3-d589-4565-9644-e6af75975c53","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3ec4f045-c714-4a22-bd1a-e07330b45ab3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"cddb9867-8e30-4951-9b13-3d1e78b9c45a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dc6d9911-785d-432f-8b53-56a1ccab302f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"77c88437-b587-496e-8b4d-1824ea8b5305","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ce963d96-5a8e-44c3-99de-c0dbebe234ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:48:34.423 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:34.440+0000 7f2f437fe640 1 -- 172.21.15.88:0/3377064795 <== mon.2 v1:172.21.15.88:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 344506374) 0x7f2f4c050ef0 con 0x7f2f5c0b1f60 2024-03-29T10:48:34.423 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:34.440+0000 7f2f437fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:48:34.423 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:34.440+0000 7f2f437fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:48:34.426 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:34.443+0000 7f2f6177a640 10 librados: watch_flush enter 2024-03-29T10:48:34.426 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:34.443+0000 7f2f6177a640 10 librados: watch_flush exit 2024-03-29T10:48:34.426 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:34.443+0000 7f2f6177a640 20 client.4544.objecter shutdown clearing up homeless session... 2024-03-29T10:48:34.426 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:34.443+0000 7f2f6177a640 10 client.4544.objecter successfully canceled tick 2024-03-29T10:48:34.426 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:34.443+0000 7f2f6177a640 1 -- 172.21.15.88:0/3377064795 >> v1:172.21.15.88:6816/1752861349 conn(0x7f2f4806dfe0 legacy=0x7f2f480704a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:34.426 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:34.443+0000 7f2f6177a640 10 monclient: shutdown 2024-03-29T10:48:34.426 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:34.443+0000 7f2f6177a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:48:34.426 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:34.443+0000 7f2f6177a640 1 -- 172.21.15.88:0/3377064795 >> v1:172.21.15.88:6790/0 conn(0x7f2f5c0b1f60 legacy=0x7f2f5c0b67d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:34.426 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:34.443+0000 7f2f6177a640 1 -- 172.21.15.88:0/3377064795 shutdown_connections 2024-03-29T10:48:34.426 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:34.443+0000 7f2f6177a640 1 -- 172.21.15.88:0/3377064795 >> 172.21.15.88:0/3377064795 conn(0x7f2f5c066a00 msgr2=0x7f2f5c153530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:48:34.427 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:34.443+0000 7f2f6177a640 1 -- 172.21.15.88:0/3377064795 shutdown_connections 2024-03-29T10:48:34.427 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:34.443+0000 7f2f6177a640 1 -- 172.21.15.88:0/3377064795 wait complete. 2024-03-29T10:48:34.427 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:34.443+0000 7f2f6177a640 1 librados: shutdown 2024-03-29T10:48:34.438 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:48:41.246 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:48:41.398 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:41.415+0000 7fba94fe8640 10 monclient: get_monmap_and_config 2024-03-29T10:48:41.398 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:41.415+0000 7fba94fe8640 10 monclient: build_initial_monmap 2024-03-29T10:48:41.399 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:41.415+0000 7fba94fe8640 10 monclient: monmap: 2024-03-29T10:48:41.399 INFO:teuthology.misc.health.smithi088.stderr:epoch 0 2024-03-29T10:48:41.399 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:48:41.399 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:48:41.416869+0000 2024-03-29T10:48:41.399 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:48:41.416869+0000 2024-03-29T10:48:41.399 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 0 (unknown) 2024-03-29T10:48:41.399 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:48:41.399 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-03-29T10:48:41.399 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-29T10:48:41.399 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-b 2024-03-29T10:48:41.399 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:48:41.401 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:41.417+0000 7fba94fe8640 1 Processor -- start 2024-03-29T10:48:41.401 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:41.417+0000 7fba94fe8640 1 -- start start 2024-03-29T10:48:41.401 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:41.417+0000 7fba94fe8640 10 monclient: init 2024-03-29T10:48:41.401 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:41.417+0000 7fba94fe8640 10 monclient: _reopen_session rank -1 2024-03-29T10:48:41.401 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:41.417+0000 7fba94fe8640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:48:41.401 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:41.417+0000 7fba94fe8640 10 monclient(hunting): picked mon.noname-b con 0x7fba900a9750 addr v1:172.21.15.88:6790/0 2024-03-29T10:48:41.401 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:41.417+0000 7fba94fe8640 10 monclient(hunting): picked mon.noname-a con 0x7fba900adca0 addr v1:172.21.15.88:6789/0 2024-03-29T10:48:41.401 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:41.418+0000 7fba94fe8640 10 monclient(hunting): picked mon.noname-c con 0x7fba90146a80 addr v1:172.21.15.196:6789/0 2024-03-29T10:48:41.402 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:41.418+0000 7fba94fe8640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba900abdc0 con 0x7fba900adca0 2024-03-29T10:48:41.402 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:41.418+0000 7fba94fe8640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba900b0070 con 0x7fba90146a80 2024-03-29T10:48:41.402 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:41.418+0000 7fba94fe8640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba900af880 con 0x7fba900a9750 2024-03-29T10:48:41.402 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:41.418+0000 7fba94fe8640 10 monclient(hunting): _renew_subs 2024-03-29T10:48:41.402 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:41.418+0000 7fba94fe8640 10 monclient(hunting): authenticate will time out at 1243.653942s 2024-03-29T10:48:41.403 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:41.421+0000 7fba8ffff640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7fba90146a80 0x7fba90151bf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.88:32823/0 (socket says 172.21.15.88:32823) 2024-03-29T10:48:41.403 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:41.421+0000 7fba8ffff640 1 -- 172.21.15.88:0/3964690918 learned_addr learned my addr 172.21.15.88:0/3964690918 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-29T10:48:41.403 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:41.421+0000 7fba8e7fc640 1 -- 172.21.15.88:0/3964690918 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2643828940 0 0) 0x7fba900b0070 con 0x7fba90146a80 2024-03-29T10:48:41.403 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:41.421+0000 7fba8e7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:48:41.403 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:41.421+0000 7fba8e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:48:41.403 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:41.421+0000 7fba8e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:48:41.403 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:41.421+0000 7fba8e7fc640 10 monclient(hunting): my global_id is 4498 2024-03-29T10:48:41.403 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:41.421+0000 7fba8e7fc640 1 -- 172.21.15.88:0/3964690918 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba78003520 con 0x7fba90146a80 2024-03-29T10:48:41.405 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:41.423+0000 7fba8e7fc640 1 -- 172.21.15.88:0/3964690918 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2340413633 0 0) 0x7fba78003520 con 0x7fba90146a80 2024-03-29T10:48:41.405 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:41.423+0000 7fba8e7fc640 1 -- 172.21.15.88:0/3964690918 >> v1:172.21.15.88:6790/0 conn(0x7fba900a9750 legacy=0x7fba90143710 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:48:41.406 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:41.424+0000 7fba8e7fc640 1 -- 172.21.15.88:0/3964690918 >> v1:172.21.15.88:6789/0 conn(0x7fba900adca0 legacy=0x7fba90146370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:41.406 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:41.424+0000 7fba8e7fc640 10 monclient: _finish_hunting 0 2024-03-29T10:48:41.407 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:41.424+0000 7fba8e7fc640 1 monclient: found mon.noname-c 2024-03-29T10:48:41.407 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:41.424+0000 7fba8e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:48:41.407 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:41.424+0000 7fba8e7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.196:6789/0 2024-03-29T10:48:41.407 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:41.424+0000 7fba8e7fc640 1 -- 172.21.15.88:0/3964690918 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba900ac030 con 0x7fba90146a80 2024-03-29T10:48:41.407 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:41.424+0000 7fba8e7fc640 10 monclient: _finish_auth 0 2024-03-29T10:48:41.407 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:41.424+0000 7fba8e7fc640 10 monclient: _check_auth_tickets 2024-03-29T10:48:41.407 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:41.424+0000 7fba8e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:48:41.407 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:41.424+0000 7fba94fe8640 5 monclient: authenticate success, global_id 4498 2024-03-29T10:48:41.408 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:41.424+0000 7fba94fe8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:48:41.408 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:41.424+0000 7fba8e7fc640 1 -- 172.21.15.88:0/3964690918 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7fba88001a10 con 0x7fba90146a80 2024-03-29T10:48:41.408 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:41.424+0000 7fba8e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:41.408 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:41.424+0000 7fba8e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:48:41.408 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:41.424+0000 7fba8e7fc640 10 monclient: dump: 2024-03-29T10:48:41.408 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:48:41.408 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:48:41.408 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:48:41.408 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:48:41.408 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:41.408 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:48:41.408 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:48:41.408 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:48:41.408 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:48:41.408 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:48:41.409 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:41.424+0000 7fba94fe8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:48:41.409 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:41.424+0000 7fba8e7fc640 1 -- 172.21.15.88:0/3964690918 <== mon.1 v1:172.21.15.196:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fba88001ca0 con 0x7fba90146a80 2024-03-29T10:48:41.409 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:41.424+0000 7fba8e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:48:41.409 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:41.424+0000 7fba94fe8640 10 monclient: get_monmap_and_config success 2024-03-29T10:48:41.409 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:41.424+0000 7fba94fe8640 4 set_mon_vals no callback set 2024-03-29T10:48:41.409 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:41.425+0000 7fba94fe8640 10 monclient: shutdown 2024-03-29T10:48:41.409 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:41.425+0000 7fba94fe8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:48:41.409 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:41.425+0000 7fba94fe8640 1 -- 172.21.15.88:0/3964690918 >> v1:172.21.15.196:6789/0 conn(0x7fba90146a80 legacy=0x7fba90151bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:41.409 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:41.426+0000 7fba94fe8640 1 -- 172.21.15.88:0/3964690918 shutdown_connections 2024-03-29T10:48:41.409 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:41.426+0000 7fba94fe8640 1 -- 172.21.15.88:0/3964690918 >> 172.21.15.88:0/3964690918 conn(0x7fba900673e0 msgr2=0x7fba900b3820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:48:41.410 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:41.428+0000 7fba94fe8640 1 -- 172.21.15.88:0/3964690918 shutdown_connections 2024-03-29T10:48:41.410 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:41.428+0000 7fba94fe8640 1 -- 172.21.15.88:0/3964690918 wait complete. 2024-03-29T10:48:41.412 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:41.429+0000 7fba94fe8640 10 monclient: build_initial_monmap 2024-03-29T10:48:41.412 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:41.429+0000 7fba94fe8640 10 monclient: monmap: 2024-03-29T10:48:41.412 INFO:teuthology.misc.health.smithi088.stderr:epoch 0 2024-03-29T10:48:41.412 INFO:teuthology.misc.health.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:48:41.412 INFO:teuthology.misc.health.smithi088.stderr:last_changed 0.000000 2024-03-29T10:48:41.412 INFO:teuthology.misc.health.smithi088.stderr:created 0.000000 2024-03-29T10:48:41.412 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 0 (unknown) 2024-03-29T10:48:41.413 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:48:41.413 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-03-29T10:48:41.413 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-29T10:48:41.413 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-c 2024-03-29T10:48:41.413 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:48:41.413 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:41.429+0000 7fba94fe8640 1 librados: starting msgr at 2024-03-29T10:48:41.413 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:41.429+0000 7fba94fe8640 1 librados: starting objecter 2024-03-29T10:48:41.413 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:41.429+0000 7fba94fe8640 1 Processor -- start 2024-03-29T10:48:41.413 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:41.430+0000 7fba94fe8640 1 -- start start 2024-03-29T10:48:41.413 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:41.430+0000 7fba94fe8640 1 librados: setting wanted keys 2024-03-29T10:48:41.413 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:41.430+0000 7fba94fe8640 1 librados: calling monclient init 2024-03-29T10:48:41.413 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:41.430+0000 7fba94fe8640 10 monclient: init 2024-03-29T10:48:41.414 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:41.430+0000 7fba94fe8640 10 monclient: _reopen_session rank -1 2024-03-29T10:48:41.414 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:41.430+0000 7fba94fe8640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:48:41.415 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:41.430+0000 7fba94fe8640 10 monclient(hunting): picked mon.noname-a con 0x7fba900a9750 addr v1:172.21.15.88:6789/0 2024-03-29T10:48:41.415 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:41.431+0000 7fba94fe8640 10 monclient(hunting): picked mon.noname-b con 0x7fba900adca0 addr v1:172.21.15.196:6789/0 2024-03-29T10:48:41.415 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:41.431+0000 7fba94fe8640 10 monclient(hunting): picked mon.noname-c con 0x7fba90146a80 addr v1:172.21.15.88:6790/0 2024-03-29T10:48:41.415 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:41.431+0000 7fba94fe8640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba901db420 con 0x7fba900a9750 2024-03-29T10:48:41.415 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:41.431+0000 7fba94fe8640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba9006e420 con 0x7fba900adca0 2024-03-29T10:48:41.415 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:41.431+0000 7fba94fe8640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba9006e690 con 0x7fba90146a80 2024-03-29T10:48:41.415 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:41.431+0000 7fba94fe8640 10 monclient(hunting): _renew_subs 2024-03-29T10:48:41.415 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:41.431+0000 7fba94fe8640 10 monclient(hunting): authenticate will time out at 1243.666951s 2024-03-29T10:48:41.416 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:41.434+0000 7fba8ffff640 1 --1- >> v1:172.21.15.88:6790/0 conn(0x7fba90146a80 0x7fba901e5cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6790/0 says I am v1:172.21.15.88:47407/0 (socket says 172.21.15.88:47407) 2024-03-29T10:48:41.416 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:41.434+0000 7fba8ffff640 1 -- 172.21.15.88:0/975536297 learned_addr learned my addr 172.21.15.88:0/975536297 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-29T10:48:41.416 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:41.434+0000 7fba77fff640 10 client.?.objecter ms_handle_connect 0x7fba90146a80 2024-03-29T10:48:41.416 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:41.434+0000 7fba77fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:48:41.417 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:41.435+0000 7fba77fff640 1 -- 172.21.15.88:0/975536297 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 257512038 0 0) 0x7fba9006e690 con 0x7fba90146a80 2024-03-29T10:48:41.418 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:41.435+0000 7fba77fff640 10 monclient(hunting): _negotiate 2024-03-29T10:48:41.418 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:41.435+0000 7fba77fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:48:41.418 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:41.435+0000 7fba77fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:48:41.418 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:41.435+0000 7fba77fff640 10 monclient(hunting): my global_id is 4547 2024-03-29T10:48:41.418 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:41.435+0000 7fba77fff640 1 -- 172.21.15.88:0/975536297 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba7c004eb0 con 0x7fba90146a80 2024-03-29T10:48:41.418 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:41.435+0000 7fba77fff640 10 client.?.objecter ms_handle_connect 0x7fba900a9750 2024-03-29T10:48:41.418 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:41.435+0000 7fba77fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:48:41.424 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:41.438+0000 7fba77fff640 1 -- 172.21.15.88:0/975536297 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3578326819 0 0) 0x7fba7c004eb0 con 0x7fba90146a80 2024-03-29T10:48:41.424 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:41.438+0000 7fba77fff640 1 -- 172.21.15.88:0/975536297 --> v1:172.21.15.88:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fba7c005300 con 0x7fba90146a80 2024-03-29T10:48:41.425 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:41.438+0000 7fba77fff640 1 -- 172.21.15.88:0/975536297 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2669334168 0 0) 0x7fba901db420 con 0x7fba900a9750 2024-03-29T10:48:41.425 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:41.438+0000 7fba77fff640 10 monclient(hunting): _negotiate 2024-03-29T10:48:41.425 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:41.438+0000 7fba77fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:48:41.425 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:41.438+0000 7fba77fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:48:41.425 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:41.438+0000 7fba77fff640 10 monclient(hunting): my global_id is 4518 2024-03-29T10:48:41.425 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:41.438+0000 7fba77fff640 1 -- 172.21.15.88:0/975536297 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba7c004eb0 con 0x7fba900a9750 2024-03-29T10:48:41.425 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:41.438+0000 7fba77fff640 1 -- 172.21.15.88:0/975536297 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7fba88004d00 con 0x7fba90146a80 2024-03-29T10:48:41.425 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:41.438+0000 7fba77fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:41.425 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:41.438+0000 7fba77fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:48:41.425 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:41.438+0000 7fba77fff640 10 monclient(hunting): dump: 2024-03-29T10:48:41.425 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:48:41.425 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:48:41.425 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:48:41.425 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:48:41.426 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:41.426 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:48:41.426 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:48:41.426 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:48:41.426 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:48:41.426 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:48:41.426 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:41.438+0000 7fba77fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:48:41.426 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:41.439+0000 7fba77fff640 10 client.?.objecter ms_handle_connect 0x7fba900adca0 2024-03-29T10:48:41.426 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:41.439+0000 7fba77fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:48:41.426 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:41.439+0000 7fba77fff640 1 -- 172.21.15.88:0/975536297 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3948155061 0 0) 0x7fba7c004eb0 con 0x7fba900a9750 2024-03-29T10:48:41.426 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:41.439+0000 7fba77fff640 1 -- 172.21.15.88:0/975536297 --> v1:172.21.15.88:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fba901db420 con 0x7fba900a9750 2024-03-29T10:48:41.426 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:41.439+0000 7fba77fff640 1 -- 172.21.15.88:0/975536297 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7fba84001730 con 0x7fba900a9750 2024-03-29T10:48:41.426 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:41.439+0000 7fba77fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:41.426 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:41.439+0000 7fba77fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:48:41.427 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:41.439+0000 7fba77fff640 10 monclient(hunting): dump: 2024-03-29T10:48:41.427 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:48:41.427 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:48:41.427 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:48:41.427 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:48:41.427 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:41.427 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:48:41.427 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:48:41.427 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:48:41.427 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:48:41.427 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:48:41.427 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:41.439+0000 7fba77fff640 1 -- 172.21.15.88:0/975536297 <== mon.2 v1:172.21.15.88:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2220835062 0 0) 0x7fba7c005300 con 0x7fba90146a80 2024-03-29T10:48:41.427 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:41.439+0000 7fba77fff640 1 -- 172.21.15.88:0/975536297 >> v1:172.21.15.196:6789/0 conn(0x7fba900adca0 legacy=0x7fba90145a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:41.427 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:41.439+0000 7fba77fff640 1 -- 172.21.15.88:0/975536297 >> v1:172.21.15.88:6789/0 conn(0x7fba900a9750 legacy=0x7fba901db8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:41.428 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:41.439+0000 7fba77fff640 10 monclient: _finish_hunting 0 2024-03-29T10:48:41.428 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:41.439+0000 7fba77fff640 1 monclient: found mon.c 2024-03-29T10:48:41.428 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:41.439+0000 7fba77fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:48:41.428 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:41.439+0000 7fba77fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-03-29T10:48:41.428 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:41.439+0000 7fba77fff640 1 -- 172.21.15.88:0/975536297 --> v1:172.21.15.88:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba901dbfc0 con 0x7fba90146a80 2024-03-29T10:48:41.428 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:41.439+0000 7fba77fff640 10 monclient: _finish_auth 0 2024-03-29T10:48:41.428 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:41.439+0000 7fba77fff640 10 monclient: _check_auth_tickets 2024-03-29T10:48:41.428 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:41.439+0000 7fba77fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:48:41.428 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:41.439+0000 7fba94fe8640 5 monclient: authenticate success, global_id 4547 2024-03-29T10:48:41.428 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:41.439+0000 7fba94fe8640 10 monclient: _renew_subs 2024-03-29T10:48:41.428 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:41.439+0000 7fba94fe8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-03-29T10:48:41.428 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:41.439+0000 7fba94fe8640 1 -- 172.21.15.88:0/975536297 --> v1:172.21.15.88:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba900b1ab0 con 0x7fba90146a80 2024-03-29T10:48:41.428 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:41.439+0000 7fba94fe8640 10 client.4547.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:48:41.428 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:41.439+0000 7fba94fe8640 10 monclient: _renew_subs 2024-03-29T10:48:41.429 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:41.439+0000 7fba94fe8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-03-29T10:48:41.429 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:41.439+0000 7fba94fe8640 1 -- 172.21.15.88:0/975536297 --> v1:172.21.15.88:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fba90146330 con 0x7fba90146a80 2024-03-29T10:48:41.429 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:41.439+0000 7fba94fe8640 1 librados: init done 2024-03-29T10:48:41.429 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:41.440+0000 7fba77fff640 1 -- 172.21.15.88:0/975536297 <== mon.2 v1:172.21.15.88:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fba88001e90 con 0x7fba90146a80 2024-03-29T10:48:41.429 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:41.440+0000 7fba77fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:48:41.429 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:41.440+0000 7fba77fff640 1 -- 172.21.15.88:0/975536297 <== mon.2 v1:172.21.15.88:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7fba880052b0 con 0x7fba90146a80 2024-03-29T10:48:41.429 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:41.440+0000 7fba77fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:41.429 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:41.440+0000 7fba77fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:48:41.429 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:41.440+0000 7fba77fff640 10 monclient: dump: 2024-03-29T10:48:41.429 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:48:41.429 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:48:41.429 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:48:41.429 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:48:41.429 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:41.429 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:48:41.430 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:48:41.430 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:48:41.430 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:48:41.430 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:48:41.430 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:41.440+0000 7fba8dffb640 4 set_mon_vals no callback set 2024-03-29T10:48:41.430 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:41.440+0000 7fba94fe8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:48:41.430 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:41.441+0000 7fba94fe8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:48:41.430 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:41.441+0000 7fba94fe8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-03-29T10:48:41.430 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:41.441+0000 7fba94fe8640 1 -- 172.21.15.88:0/975536297 --> v1:172.21.15.88:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba70005080 con 0x7fba90146a80 2024-03-29T10:48:41.431 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:41.443+0000 7fba77fff640 1 -- 172.21.15.88:0/975536297 <== mon.2 v1:172.21.15.88:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4235629220 0 0) 0x7fba88005410 con 0x7fba90146a80 2024-03-29T10:48:41.431 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:41.444+0000 7fba77fff640 1 -- 172.21.15.88:0/975536297 <== mon.2 v1:172.21.15.88:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4257673582 0 0) 0x7fba88054d60 con 0x7fba90146a80 2024-03-29T10:48:41.431 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:41.444+0000 7fba77fff640 10 client.4547.objecter ms_dispatch 0x7fba90000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:48:41.431 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:41.444+0000 7fba77fff640 3 client.4547.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:48:41.431 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:41.444+0000 7fba77fff640 3 client.4547.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:48:41.432 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:41.444+0000 7fba77fff640 20 client.4547.objecter dump_active .. 0 homeless 2024-03-29T10:48:41.432 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:41.445+0000 7fba8f7fe640 10 monclient: get_auth_request con 0x7fba7c06e130 auth_method 0 2024-03-29T10:48:41.432 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:41.446+0000 7fba77fff640 10 client.4547.objecter ms_handle_connect 0x7fba7c06e130 2024-03-29T10:48:41.436 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:41.454+0000 7fba77fff640 1 -- 172.21.15.88:0/975536297 <== mon.2 v1:172.21.15.88:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fba88051000 con 0x7fba90146a80 2024-03-29T10:48:41.436 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:41.454+0000 7fba77fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:48:41.436 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:41.454+0000 7fba77fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:48:41.762 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:41.780+0000 7fba94fe8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:48:41.763 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:41.780+0000 7fba94fe8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:48:41.763 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:41.780+0000 7fba94fe8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-03-29T10:48:41.763 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:41.780+0000 7fba94fe8640 1 -- 172.21.15.88:0/975536297 --> v1:172.21.15.88:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fba70005330 con 0x7fba90146a80 2024-03-29T10:48:41.766 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:41.783+0000 7fba77fff640 1 -- 172.21.15.88:0/975536297 <== mon.2 v1:172.21.15.88:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 344506374) 0x7fba880511a0 con 0x7fba90146a80 2024-03-29T10:48:41.766 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:41.783+0000 7fba77fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:48:41.766 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:41.783+0000 7fba77fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:48:41.766 INFO:teuthology.misc.health.smithi088.stdout: 2024-03-29T10:48:41.767 INFO:teuthology.misc.health.smithi088.stdout:{"epoch":10,"fsid":"869f5188-f92b-42dd-a46d-8c646e0388a9","created":"2024-03-29T10:40:40.285087+0000","modified":"2024-03-29T10:40:42.536230+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:40:42.536230+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a26c0c1c-8de0-438c-a3f6-3df10ea61574","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5de24915-8dec-49b6-8b16-f3170197c591","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ac054af3-d589-4565-9644-e6af75975c53","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3ec4f045-c714-4a22-bd1a-e07330b45ab3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"cddb9867-8e30-4951-9b13-3d1e78b9c45a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dc6d9911-785d-432f-8b53-56a1ccab302f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"77c88437-b587-496e-8b4d-1824ea8b5305","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ce963d96-5a8e-44c3-99de-c0dbebe234ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:48:41.772 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:41.789+0000 7fba94fe8640 10 librados: watch_flush enter 2024-03-29T10:48:41.772 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:41.790+0000 7fba94fe8640 10 librados: watch_flush exit 2024-03-29T10:48:41.772 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:41.790+0000 7fba94fe8640 20 client.4547.objecter shutdown clearing up homeless session... 2024-03-29T10:48:41.772 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:41.790+0000 7fba94fe8640 10 client.4547.objecter successfully canceled tick 2024-03-29T10:48:41.773 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:41.790+0000 7fba94fe8640 1 -- 172.21.15.88:0/975536297 >> v1:172.21.15.88:6816/1752861349 conn(0x7fba7c06e130 legacy=0x7fba7c0705f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:41.773 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:41.790+0000 7fba94fe8640 10 monclient: shutdown 2024-03-29T10:48:41.773 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:41.790+0000 7fba94fe8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:48:41.773 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:41.790+0000 7fba94fe8640 1 -- 172.21.15.88:0/975536297 >> v1:172.21.15.88:6790/0 conn(0x7fba90146a80 legacy=0x7fba901e5cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:41.773 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:41.790+0000 7fba94fe8640 1 -- 172.21.15.88:0/975536297 shutdown_connections 2024-03-29T10:48:41.773 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:41.790+0000 7fba94fe8640 1 -- 172.21.15.88:0/975536297 >> 172.21.15.88:0/975536297 conn(0x7fba900673e0 msgr2=0x7fba900ae420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:48:41.773 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:41.791+0000 7fba94fe8640 1 -- 172.21.15.88:0/975536297 shutdown_connections 2024-03-29T10:48:41.774 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:41.791+0000 7fba94fe8640 1 -- 172.21.15.88:0/975536297 wait complete. 2024-03-29T10:48:41.774 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:41.791+0000 7fba94fe8640 1 librados: shutdown 2024-03-29T10:48:41.790 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:48:48.597 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:48:48.744 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:48.761+0000 7efccb3d3640 10 monclient: get_monmap_and_config 2024-03-29T10:48:48.744 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:48.761+0000 7efccb3d3640 10 monclient: build_initial_monmap 2024-03-29T10:48:48.744 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:48.761+0000 7efccb3d3640 10 monclient: monmap: 2024-03-29T10:48:48.744 INFO:teuthology.misc.health.smithi088.stderr:epoch 0 2024-03-29T10:48:48.744 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:48:48.744 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:48:48.763010+0000 2024-03-29T10:48:48.744 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:48:48.763010+0000 2024-03-29T10:48:48.744 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 0 (unknown) 2024-03-29T10:48:48.744 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:48:48.744 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-03-29T10:48:48.744 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-29T10:48:48.744 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-b 2024-03-29T10:48:48.745 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:48:48.748 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:48.763+0000 7efccb3d3640 1 Processor -- start 2024-03-29T10:48:48.748 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:48.763+0000 7efccb3d3640 1 -- start start 2024-03-29T10:48:48.748 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:48.763+0000 7efccb3d3640 10 monclient: init 2024-03-29T10:48:48.748 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:48.764+0000 7efccb3d3640 10 monclient: _reopen_session rank -1 2024-03-29T10:48:48.748 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:48.764+0000 7efccb3d3640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:48:48.748 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:48.764+0000 7efccb3d3640 10 monclient(hunting): picked mon.noname-a con 0x7efcbc0389e0 addr v1:172.21.15.88:6789/0 2024-03-29T10:48:48.748 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:48.764+0000 7efccb3d3640 10 monclient(hunting): picked mon.noname-b con 0x7efcbc038dc0 addr v1:172.21.15.88:6790/0 2024-03-29T10:48:48.748 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:48.764+0000 7efccb3d3640 10 monclient(hunting): picked mon.noname-c con 0x7efcbc030660 addr v1:172.21.15.196:6789/0 2024-03-29T10:48:48.748 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:48.764+0000 7efccb3d3640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efcbc0c54c0 con 0x7efcbc0389e0 2024-03-29T10:48:48.748 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:48.764+0000 7efccb3d3640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efcbc036430 con 0x7efcbc030660 2024-03-29T10:48:48.748 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:48.764+0000 7efccb3d3640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efcbc039840 con 0x7efcbc038dc0 2024-03-29T10:48:48.748 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:48.764+0000 7efccb3d3640 10 monclient(hunting): _renew_subs 2024-03-29T10:48:48.749 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:48.764+0000 7efccb3d3640 10 monclient(hunting): authenticate will time out at 1251.000071s 2024-03-29T10:48:48.752 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:48.770+0000 7efccabd2640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7efcbc030660 0x7efcbc0d3170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.88:53311/0 (socket says 172.21.15.88:53311) 2024-03-29T10:48:48.752 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:48.770+0000 7efccabd2640 1 -- 172.21.15.88:0/3125678659 learned_addr learned my addr 172.21.15.88:0/3125678659 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-29T10:48:48.753 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:48.771+0000 7efcc93cf640 1 -- 172.21.15.88:0/3125678659 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 621754416 0 0) 0x7efcbc036430 con 0x7efcbc030660 2024-03-29T10:48:48.753 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:48.771+0000 7efcc93cf640 10 monclient(hunting): _negotiate 2024-03-29T10:48:48.754 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:48.771+0000 7efcc93cf640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:48:48.754 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:48.771+0000 7efcc93cf640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:48:48.754 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:48.771+0000 7efcc93cf640 10 monclient(hunting): my global_id is 4504 2024-03-29T10:48:48.754 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:48.771+0000 7efcc93cf640 1 -- 172.21.15.88:0/3125678659 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efcb0003520 con 0x7efcbc030660 2024-03-29T10:48:48.755 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:48.773+0000 7efcc93cf640 1 -- 172.21.15.88:0/3125678659 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3167873798 0 0) 0x7efcb0003520 con 0x7efcbc030660 2024-03-29T10:48:48.755 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:48.773+0000 7efcc93cf640 1 -- 172.21.15.88:0/3125678659 >> v1:172.21.15.88:6790/0 conn(0x7efcbc038dc0 legacy=0x7efcbc0cadd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:48:48.757 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:48.773+0000 7efcc93cf640 1 -- 172.21.15.88:0/3125678659 >> v1:172.21.15.88:6789/0 conn(0x7efcbc0389e0 legacy=0x7efcbc033520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:48:48.757 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:48.773+0000 7efcc93cf640 10 monclient: _finish_hunting 0 2024-03-29T10:48:48.757 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:48.773+0000 7efcc93cf640 1 monclient: found mon.noname-c 2024-03-29T10:48:48.757 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:48.773+0000 7efcc93cf640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:48:48.757 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:48.773+0000 7efcc93cf640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.196:6789/0 2024-03-29T10:48:48.757 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:48.773+0000 7efcc93cf640 1 -- 172.21.15.88:0/3125678659 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcbc035690 con 0x7efcbc030660 2024-03-29T10:48:48.757 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:48.773+0000 7efcc93cf640 10 monclient: _finish_auth 0 2024-03-29T10:48:48.757 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:48.773+0000 7efcc93cf640 10 monclient: _check_auth_tickets 2024-03-29T10:48:48.757 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:48.773+0000 7efcc93cf640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:48:48.757 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:48.773+0000 7efccb3d3640 5 monclient: authenticate success, global_id 4504 2024-03-29T10:48:48.757 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:48.773+0000 7efccb3d3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:48:48.758 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:48.773+0000 7efcc93cf640 1 -- 172.21.15.88:0/3125678659 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7efcc404f730 con 0x7efcbc030660 2024-03-29T10:48:48.758 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:48.773+0000 7efcc93cf640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:48.758 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:48.773+0000 7efcc93cf640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:48:48.758 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:48.773+0000 7efcc93cf640 10 monclient: dump: 2024-03-29T10:48:48.758 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:48:48.758 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:48:48.758 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:48:48.758 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:48:48.758 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:48.758 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:48:48.758 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:48:48.758 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:48:48.758 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:48:48.759 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:48:48.759 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:48.773+0000 7efccb3d3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:48:48.759 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:48.774+0000 7efcc93cf640 1 -- 172.21.15.88:0/3125678659 <== mon.1 v1:172.21.15.196:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efcc4075f10 con 0x7efcbc030660 2024-03-29T10:48:48.759 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:48.774+0000 7efcc93cf640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:48:48.759 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:48.774+0000 7efccb3d3640 10 monclient: get_monmap_and_config success 2024-03-29T10:48:48.759 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:48.774+0000 7efccb3d3640 4 set_mon_vals no callback set 2024-03-29T10:48:48.759 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:48.774+0000 7efcc93cf640 1 -- 172.21.15.88:0/3125678659 <== mon.1 v1:172.21.15.196:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7efcc40758b0 con 0x7efcbc030660 2024-03-29T10:48:48.759 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:48.774+0000 7efcc93cf640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:48.759 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:48.774+0000 7efcc93cf640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:48:48.759 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:48.774+0000 7efcc93cf640 10 monclient: dump: 2024-03-29T10:48:48.759 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:48:48.759 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:48:48.759 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:48:48.759 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:48:48.759 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:48.760 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:48:48.760 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:48:48.760 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:48:48.760 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:48:48.760 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:48:48.760 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:48.774+0000 7efccb3d3640 10 monclient: shutdown 2024-03-29T10:48:48.760 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:48.774+0000 7efccb3d3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:48:48.760 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:48.774+0000 7efccb3d3640 1 -- 172.21.15.88:0/3125678659 >> v1:172.21.15.196:6789/0 conn(0x7efcbc030660 legacy=0x7efcbc0d3170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:48.761 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:48.775+0000 7efccb3d3640 1 -- 172.21.15.88:0/3125678659 shutdown_connections 2024-03-29T10:48:48.761 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:48.775+0000 7efccb3d3640 1 -- 172.21.15.88:0/3125678659 >> 172.21.15.88:0/3125678659 conn(0x7efcbc005c00 msgr2=0x7efcbc006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:48:48.761 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:48.776+0000 7efccb3d3640 1 -- 172.21.15.88:0/3125678659 shutdown_connections 2024-03-29T10:48:48.761 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:48.778+0000 7efccb3d3640 1 -- 172.21.15.88:0/3125678659 wait complete. 2024-03-29T10:48:48.761 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:48.779+0000 7efccb3d3640 10 monclient: build_initial_monmap 2024-03-29T10:48:48.761 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:48.779+0000 7efccb3d3640 10 monclient: monmap: 2024-03-29T10:48:48.761 INFO:teuthology.misc.health.smithi088.stderr:epoch 0 2024-03-29T10:48:48.761 INFO:teuthology.misc.health.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:48:48.761 INFO:teuthology.misc.health.smithi088.stderr:last_changed 0.000000 2024-03-29T10:48:48.761 INFO:teuthology.misc.health.smithi088.stderr:created 0.000000 2024-03-29T10:48:48.762 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 0 (unknown) 2024-03-29T10:48:48.762 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:48:48.762 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-03-29T10:48:48.762 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-29T10:48:48.762 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-c 2024-03-29T10:48:48.762 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:48:48.763 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:48.779+0000 7efccb3d3640 1 librados: starting msgr at 2024-03-29T10:48:48.763 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:48.779+0000 7efccb3d3640 1 librados: starting objecter 2024-03-29T10:48:48.763 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:48.779+0000 7efccb3d3640 1 Processor -- start 2024-03-29T10:48:48.763 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:48.779+0000 7efccb3d3640 1 -- start start 2024-03-29T10:48:48.763 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:48.779+0000 7efccb3d3640 1 librados: setting wanted keys 2024-03-29T10:48:48.763 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:48.779+0000 7efccb3d3640 1 librados: calling monclient init 2024-03-29T10:48:48.763 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:48.779+0000 7efccb3d3640 10 monclient: init 2024-03-29T10:48:48.763 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:48.780+0000 7efccb3d3640 10 monclient: _reopen_session rank -1 2024-03-29T10:48:48.764 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:48.780+0000 7efccb3d3640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:48:48.764 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:48.780+0000 7efccb3d3640 10 monclient(hunting): picked mon.noname-a con 0x7efcbc030660 addr v1:172.21.15.88:6789/0 2024-03-29T10:48:48.764 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:48.780+0000 7efccb3d3640 10 monclient(hunting): picked mon.noname-b con 0x7efcbc0389e0 addr v1:172.21.15.196:6789/0 2024-03-29T10:48:48.764 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:48.780+0000 7efccb3d3640 10 monclient(hunting): picked mon.noname-c con 0x7efcbc164150 addr v1:172.21.15.88:6790/0 2024-03-29T10:48:48.764 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:48.780+0000 7efccb3d3640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efcbc00a760 con 0x7efcbc030660 2024-03-29T10:48:48.764 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:48.780+0000 7efccb3d3640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efcbc0c36d0 con 0x7efcbc0389e0 2024-03-29T10:48:48.764 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:48.780+0000 7efccb3d3640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efcbc00aa40 con 0x7efcbc164150 2024-03-29T10:48:48.764 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:48.780+0000 7efccb3d3640 10 monclient(hunting): _renew_subs 2024-03-29T10:48:48.764 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:48.780+0000 7efccb3d3640 10 monclient(hunting): authenticate will time out at 1251.016627s 2024-03-29T10:48:48.766 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:48.784+0000 7efcca3d1640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7efcbc030660 0x7efcbc02f8a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:41849/0 (socket says 172.21.15.88:41849) 2024-03-29T10:48:48.766 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:48.784+0000 7efcca3d1640 1 -- 172.21.15.88:0/1933112919 learned_addr learned my addr 172.21.15.88:0/1933112919 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-29T10:48:48.766 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:48.784+0000 7efcbaffd640 10 client.?.objecter ms_handle_connect 0x7efcbc030660 2024-03-29T10:48:48.766 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:48.784+0000 7efcbaffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:48:48.768 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:48.786+0000 7efcbaffd640 1 -- 172.21.15.88:0/1933112919 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2517043201 0 0) 0x7efcbc00a760 con 0x7efcbc030660 2024-03-29T10:48:48.768 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:48.786+0000 7efcbaffd640 10 monclient(hunting): _negotiate 2024-03-29T10:48:48.768 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:48.786+0000 7efcbaffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:48:48.768 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:48.786+0000 7efcbaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:48:48.768 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:48.786+0000 7efcbaffd640 10 monclient(hunting): my global_id is 4521 2024-03-29T10:48:48.768 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:48.786+0000 7efcbaffd640 1 -- 172.21.15.88:0/1933112919 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efca8004eb0 con 0x7efcbc030660 2024-03-29T10:48:48.768 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:48.786+0000 7efcbaffd640 1 -- 172.21.15.88:0/1933112919 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1643982860 0 0) 0x7efca8004eb0 con 0x7efcbc030660 2024-03-29T10:48:48.769 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:48.786+0000 7efcbaffd640 1 -- 172.21.15.88:0/1933112919 --> v1:172.21.15.88:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efcbc00a760 con 0x7efcbc030660 2024-03-29T10:48:48.769 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:48.787+0000 7efcbaffd640 1 -- 172.21.15.88:0/1933112919 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7efcc0004080 con 0x7efcbc030660 2024-03-29T10:48:48.769 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:48.787+0000 7efcbaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:48.770 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:48.787+0000 7efcbaffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:48:48.770 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:48.787+0000 7efcbaffd640 10 monclient(hunting): dump: 2024-03-29T10:48:48.770 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:48:48.771 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:48:48.771 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:48:48.771 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:48:48.771 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:48.771 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:48:48.771 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:48:48.771 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:48:48.771 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:48:48.771 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:48:48.771 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:48.787+0000 7efcbaffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:48:48.771 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:48.787+0000 7efcbaffd640 1 -- 172.21.15.88:0/1933112919 <== mon.0 v1:172.21.15.88:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1174167927 0 0) 0x7efcbc00a760 con 0x7efcbc030660 2024-03-29T10:48:48.771 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:48.787+0000 7efcbaffd640 1 -- 172.21.15.88:0/1933112919 >> v1:172.21.15.88:6790/0 conn(0x7efcbc164150 legacy=0x7efcbc0c2fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:48.771 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:48.787+0000 7efcbaffd640 1 -- 172.21.15.88:0/1933112919 >> v1:172.21.15.196:6789/0 conn(0x7efcbc0389e0 legacy=0x7efcbc0c28b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:48:48.771 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:48.787+0000 7efcbaffd640 10 monclient: _finish_hunting 0 2024-03-29T10:48:48.772 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:48.787+0000 7efcbaffd640 1 monclient: found mon.a 2024-03-29T10:48:48.772 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:48.787+0000 7efcbaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:48:48.772 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:48.787+0000 7efcbaffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-03-29T10:48:48.772 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:48.787+0000 7efcbaffd640 1 -- 172.21.15.88:0/1933112919 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcbc02f310 con 0x7efcbc030660 2024-03-29T10:48:48.772 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:48.787+0000 7efcbaffd640 10 monclient: _finish_auth 0 2024-03-29T10:48:48.773 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:48.787+0000 7efcbaffd640 10 monclient: _check_auth_tickets 2024-03-29T10:48:48.773 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:48.787+0000 7efcbaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:48:48.773 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:48.789+0000 7efccb3d3640 5 monclient: authenticate success, global_id 4521 2024-03-29T10:48:48.773 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:48.789+0000 7efccb3d3640 10 monclient: _renew_subs 2024-03-29T10:48:48.773 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:48.789+0000 7efccb3d3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-03-29T10:48:48.773 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:48.789+0000 7efccb3d3640 1 -- 172.21.15.88:0/1933112919 --> v1:172.21.15.88:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efcbc02ffb0 con 0x7efcbc030660 2024-03-29T10:48:48.773 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:48.789+0000 7efccb3d3640 10 client.4521.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:48:48.773 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:48.789+0000 7efccb3d3640 10 monclient: _renew_subs 2024-03-29T10:48:48.773 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:48.789+0000 7efccb3d3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-03-29T10:48:48.773 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:48.789+0000 7efccb3d3640 1 -- 172.21.15.88:0/1933112919 --> v1:172.21.15.88:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7efcbc009770 con 0x7efcbc030660 2024-03-29T10:48:48.773 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:48.789+0000 7efccb3d3640 1 librados: init done 2024-03-29T10:48:48.774 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:48.789+0000 7efcbaffd640 1 -- 172.21.15.88:0/1933112919 <== mon.0 v1:172.21.15.88:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efcc00043a0 con 0x7efcbc030660 2024-03-29T10:48:48.774 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:48.789+0000 7efcbaffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:48:48.774 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:48.789+0000 7efcbaffd640 1 -- 172.21.15.88:0/1933112919 <== mon.0 v1:172.21.15.88:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7efcc0004080 con 0x7efcbc030660 2024-03-29T10:48:48.774 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:48.789+0000 7efcbaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:48.774 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:48.789+0000 7efcbaffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:48:48.774 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:48.789+0000 7efcbaffd640 10 monclient: dump: 2024-03-29T10:48:48.774 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:48:48.774 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:48:48.774 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:48:48.774 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:48:48.774 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:48.774 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:48:48.774 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:48:48.774 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:48:48.775 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:48:48.775 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:48:48.776 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:48.791+0000 7efccb3d3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:48:48.776 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:48.791+0000 7efccb3d3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:48:48.776 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:48.791+0000 7efccb3d3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-03-29T10:48:48.776 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:48.791+0000 7efccb3d3640 1 -- 172.21.15.88:0/1933112919 --> v1:172.21.15.88:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efcbc007b90 con 0x7efcbc030660 2024-03-29T10:48:48.776 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:48.791+0000 7efcbbfff640 4 set_mon_vals no callback set 2024-03-29T10:48:48.776 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:48.791+0000 7efcbaffd640 1 -- 172.21.15.88:0/1933112919 <== mon.0 v1:172.21.15.88:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4235629220 0 0) 0x7efcc0001180 con 0x7efcbc030660 2024-03-29T10:48:48.776 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:48.792+0000 7efcbaffd640 1 -- 172.21.15.88:0/1933112919 <== mon.0 v1:172.21.15.88:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4257673582 0 0) 0x7efcc00846b0 con 0x7efcbc030660 2024-03-29T10:48:48.776 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:48.792+0000 7efcbaffd640 10 client.4521.objecter ms_dispatch 0x7efcc4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:48:48.777 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:48.792+0000 7efcbaffd640 3 client.4521.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:48:48.777 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:48.792+0000 7efcbaffd640 3 client.4521.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:48:48.777 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:48.792+0000 7efcbaffd640 20 client.4521.objecter dump_active .. 0 homeless 2024-03-29T10:48:48.777 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:48.793+0000 7efcc9bd0640 10 monclient: get_auth_request con 0x7efca806df70 auth_method 0 2024-03-29T10:48:48.777 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:48.793+0000 7efcbaffd640 10 client.4521.objecter ms_handle_connect 0x7efca806df70 2024-03-29T10:48:48.785 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:48.802+0000 7efcbaffd640 1 -- 172.21.15.88:0/1933112919 <== mon.0 v1:172.21.15.88:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7efcc004fff0 con 0x7efcbc030660 2024-03-29T10:48:48.786 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:48.802+0000 7efcbaffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:48:48.786 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:48.802+0000 7efcbaffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:48:49.109 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:49.126+0000 7efccb3d3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:48:49.109 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:49.126+0000 7efccb3d3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:48:49.109 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:49.126+0000 7efccb3d3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-03-29T10:48:49.109 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:49.126+0000 7efccb3d3640 1 -- 172.21.15.88:0/1933112919 --> v1:172.21.15.88:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7efcbc02f110 con 0x7efcbc030660 2024-03-29T10:48:49.112 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:49.128+0000 7efcbaffd640 1 -- 172.21.15.88:0/1933112919 <== mon.0 v1:172.21.15.88:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 344506374) 0x7efcc004fff0 con 0x7efcbc030660 2024-03-29T10:48:49.112 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:49.128+0000 7efcbaffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:48:49.112 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:49.128+0000 7efcbaffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:48:49.112 INFO:teuthology.misc.health.smithi088.stdout: 2024-03-29T10:48:49.112 INFO:teuthology.misc.health.smithi088.stdout:{"epoch":10,"fsid":"869f5188-f92b-42dd-a46d-8c646e0388a9","created":"2024-03-29T10:40:40.285087+0000","modified":"2024-03-29T10:40:42.536230+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:40:42.536230+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a26c0c1c-8de0-438c-a3f6-3df10ea61574","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5de24915-8dec-49b6-8b16-f3170197c591","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ac054af3-d589-4565-9644-e6af75975c53","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3ec4f045-c714-4a22-bd1a-e07330b45ab3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"cddb9867-8e30-4951-9b13-3d1e78b9c45a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dc6d9911-785d-432f-8b53-56a1ccab302f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"77c88437-b587-496e-8b4d-1824ea8b5305","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ce963d96-5a8e-44c3-99de-c0dbebe234ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:48:49.116 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:49.133+0000 7efccb3d3640 10 librados: watch_flush enter 2024-03-29T10:48:49.116 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:49.133+0000 7efccb3d3640 10 librados: watch_flush exit 2024-03-29T10:48:49.116 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:49.133+0000 7efccb3d3640 20 client.4521.objecter shutdown clearing up homeless session... 2024-03-29T10:48:49.116 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:49.133+0000 7efccb3d3640 10 client.4521.objecter successfully canceled tick 2024-03-29T10:48:49.116 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:49.133+0000 7efccb3d3640 1 -- 172.21.15.88:0/1933112919 >> v1:172.21.15.88:6816/1752861349 conn(0x7efca806df70 legacy=0x7efca8070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:49.116 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:49.133+0000 7efccb3d3640 10 monclient: shutdown 2024-03-29T10:48:49.116 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:49.134+0000 7efccb3d3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:48:49.116 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:49.134+0000 7efccb3d3640 1 -- 172.21.15.88:0/1933112919 >> v1:172.21.15.88:6789/0 conn(0x7efcbc030660 legacy=0x7efcbc02f8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:49.116 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:49.134+0000 7efccb3d3640 1 -- 172.21.15.88:0/1933112919 shutdown_connections 2024-03-29T10:48:49.116 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:49.134+0000 7efccb3d3640 1 -- 172.21.15.88:0/1933112919 >> 172.21.15.88:0/1933112919 conn(0x7efcbc005c00 msgr2=0x7efcbc034090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:48:49.117 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:49.134+0000 7efccb3d3640 1 -- 172.21.15.88:0/1933112919 shutdown_connections 2024-03-29T10:48:49.117 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:49.134+0000 7efccb3d3640 1 -- 172.21.15.88:0/1933112919 wait complete. 2024-03-29T10:48:49.117 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:49.134+0000 7efccb3d3640 1 librados: shutdown 2024-03-29T10:48:49.132 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:48:55.939 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:48:56.083 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:56.101+0000 7f083e5b5640 10 monclient: get_monmap_and_config 2024-03-29T10:48:56.083 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:56.101+0000 7f083e5b5640 10 monclient: build_initial_monmap 2024-03-29T10:48:56.083 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:56.101+0000 7f083e5b5640 10 monclient: monmap: 2024-03-29T10:48:56.083 INFO:teuthology.misc.health.smithi088.stderr:epoch 0 2024-03-29T10:48:56.084 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:48:56.084 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:48:56.102355+0000 2024-03-29T10:48:56.084 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:48:56.102355+0000 2024-03-29T10:48:56.084 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 0 (unknown) 2024-03-29T10:48:56.084 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:48:56.084 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-03-29T10:48:56.084 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-29T10:48:56.084 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-b 2024-03-29T10:48:56.084 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:48:56.087 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:56.103+0000 7f083e5b5640 1 Processor -- start 2024-03-29T10:48:56.088 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:56.106+0000 7f083e5b5640 1 -- start start 2024-03-29T10:48:56.088 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:56.106+0000 7f083e5b5640 10 monclient: init 2024-03-29T10:48:56.089 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:56.107+0000 7f083e5b5640 10 monclient: _reopen_session rank -1 2024-03-29T10:48:56.089 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:56.107+0000 7f083e5b5640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:48:56.092 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:56.110+0000 7f083e5b5640 10 monclient(hunting): picked mon.noname-c con 0x7f08381457f0 addr v1:172.21.15.196:6789/0 2024-03-29T10:48:56.097 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:56.111+0000 7f083e5b5640 10 monclient(hunting): picked mon.noname-a con 0x7f083813f650 addr v1:172.21.15.88:6789/0 2024-03-29T10:48:56.098 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:56.111+0000 7f083e5b5640 10 monclient(hunting): picked mon.noname-b con 0x7f08380a9d70 addr v1:172.21.15.88:6790/0 2024-03-29T10:48:56.098 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:56.111+0000 7f083e5b5640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08380aaef0 con 0x7f083813f650 2024-03-29T10:48:56.098 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:56.111+0000 7f083e5b5640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0838145bd0 con 0x7f08381457f0 2024-03-29T10:48:56.098 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:56.111+0000 7f083e5b5640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0838145eb0 con 0x7f08380a9d70 2024-03-29T10:48:56.098 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:56.111+0000 7f083e5b5640 10 monclient(hunting): _renew_subs 2024-03-29T10:48:56.098 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:56.111+0000 7f083e5b5640 10 monclient(hunting): authenticate will time out at 1258.347008s 2024-03-29T10:48:56.098 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:56.114+0000 7f0837fff640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f08381457f0 0x7f083813ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.88:37781/0 (socket says 172.21.15.88:37781) 2024-03-29T10:48:56.098 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:56.114+0000 7f0837fff640 1 -- 172.21.15.88:0/3593719351 learned_addr learned my addr 172.21.15.88:0/3593719351 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-29T10:48:56.098 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:56.114+0000 7f08377fe640 1 -- 172.21.15.88:0/3593719351 >> v1:172.21.15.88:6789/0 conn(0x7f083813f650 legacy=0x7f08380b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2024-03-29T10:48:56.098 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:56.114+0000 7f08377fe640 1 -- 172.21.15.88:0/3593719351 >> v1:172.21.15.88:6789/0 conn(0x7f083813f650 legacy=0x7f08380b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-29T10:48:56.098 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:56.114+0000 7f08377fe640 1 --1- 172.21.15.88:0/3593719351 >> v1:172.21.15.88:6789/0 conn(0x7f083813f650 0x7f08380b2620 :-1 s=OPENED pgs=820864 cs=1 l=1).handle_message read tag failed 2024-03-29T10:48:56.098 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:56.114+0000 7f08377fe640 1 --1- 172.21.15.88:0/3593719351 >> v1:172.21.15.88:6789/0 conn(0x7f083813f650 0x7f08380b2620 :-1 s=OPENED pgs=820864 cs=1 l=1).fault on lossy channel, failing 2024-03-29T10:48:56.099 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:56.114+0000 7f0836ffd640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.88:6789/0 2024-03-29T10:48:56.099 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:56.114+0000 7f0836ffd640 1 -- 172.21.15.88:0/3593719351 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1196544464 0 0) 0x7f0838145bd0 con 0x7f08381457f0 2024-03-29T10:48:56.099 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:56.114+0000 7f0836ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:48:56.099 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:56.114+0000 7f0836ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:48:56.099 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:56.114+0000 7f0836ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:48:56.099 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:56.114+0000 7f0836ffd640 10 monclient(hunting): my global_id is 4507 2024-03-29T10:48:56.099 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:56.114+0000 7f0836ffd640 1 -- 172.21.15.88:0/3593719351 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0820003520 con 0x7f08381457f0 2024-03-29T10:48:56.100 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:56.116+0000 7f0836ffd640 1 -- 172.21.15.88:0/3593719351 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 30176491 0 0) 0x7f0820003520 con 0x7f08381457f0 2024-03-29T10:48:56.101 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:56.116+0000 7f0836ffd640 1 -- 172.21.15.88:0/3593719351 >> v1:172.21.15.88:6790/0 conn(0x7f08380a9d70 legacy=0x7f083814d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:48:56.101 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:56.116+0000 7f0836ffd640 1 -- 172.21.15.88:0/3593719351 >> v1:172.21.15.88:6789/0 conn(0x7f083813f650 legacy=0x7f08380b2620 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-29T10:48:56.101 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:56.116+0000 7f0836ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:48:56.101 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:56.116+0000 7f0836ffd640 1 monclient: found mon.noname-c 2024-03-29T10:48:56.101 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:56.116+0000 7f0836ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:48:56.101 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:56.116+0000 7f0836ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.196:6789/0 2024-03-29T10:48:56.101 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:56.116+0000 7f0836ffd640 1 -- 172.21.15.88:0/3593719351 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0838146340 con 0x7f08381457f0 2024-03-29T10:48:56.101 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:56.116+0000 7f0836ffd640 10 monclient: _finish_auth 0 2024-03-29T10:48:56.101 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:56.116+0000 7f0836ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:48:56.101 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:56.116+0000 7f0836ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:48:56.101 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:56.116+0000 7f0836ffd640 1 -- 172.21.15.88:0/3593719351 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f0830001a10 con 0x7f08381457f0 2024-03-29T10:48:56.101 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:56.116+0000 7f0836ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:56.101 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:56.116+0000 7f0836ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:48:56.102 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:56.116+0000 7f0836ffd640 10 monclient: dump: 2024-03-29T10:48:56.102 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:48:56.102 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:48:56.102 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:48:56.102 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:48:56.102 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:56.102 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:48:56.102 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:48:56.102 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:48:56.102 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:48:56.102 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:48:56.102 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:56.117+0000 7f083e5b5640 5 monclient: authenticate success, global_id 4507 2024-03-29T10:48:56.102 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:56.117+0000 7f083e5b5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:48:56.103 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:56.117+0000 7f0836ffd640 1 -- 172.21.15.88:0/3593719351 <== mon.1 v1:172.21.15.196:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0830001b70 con 0x7f08381457f0 2024-03-29T10:48:56.103 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:56.117+0000 7f0836ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:48:56.103 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:56.117+0000 7f0836ffd640 1 -- 172.21.15.88:0/3593719351 <== mon.1 v1:172.21.15.196:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f0830001fc0 con 0x7f08381457f0 2024-03-29T10:48:56.103 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:56.117+0000 7f0836ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:56.103 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:56.117+0000 7f0836ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:48:56.103 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:56.117+0000 7f0836ffd640 10 monclient: dump: 2024-03-29T10:48:56.103 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:48:56.103 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:48:56.103 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:48:56.103 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:48:56.103 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:56.103 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:48:56.103 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:48:56.103 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:48:56.104 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:48:56.104 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:48:56.104 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:56.117+0000 7f083e5b5640 10 monclient: get_monmap_and_config success 2024-03-29T10:48:56.104 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:56.117+0000 7f083e5b5640 4 set_mon_vals no callback set 2024-03-29T10:48:56.104 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:56.118+0000 7f083e5b5640 10 monclient: shutdown 2024-03-29T10:48:56.104 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:56.118+0000 7f083e5b5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:48:56.104 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:56.118+0000 7f083e5b5640 1 -- 172.21.15.88:0/3593719351 >> v1:172.21.15.196:6789/0 conn(0x7f08381457f0 legacy=0x7f083813ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:56.104 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:56.118+0000 7f083e5b5640 1 -- 172.21.15.88:0/3593719351 shutdown_connections 2024-03-29T10:48:56.104 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:56.118+0000 7f083e5b5640 1 -- 172.21.15.88:0/3593719351 >> 172.21.15.88:0/3593719351 conn(0x7f0838066a00 msgr2=0x7f0838066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:48:56.105 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:56.121+0000 7f083e5b5640 1 -- 172.21.15.88:0/3593719351 shutdown_connections 2024-03-29T10:48:56.105 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:56.121+0000 7f083e5b5640 1 -- 172.21.15.88:0/3593719351 wait complete. 2024-03-29T10:48:56.105 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:56.122+0000 7f083e5b5640 10 monclient: build_initial_monmap 2024-03-29T10:48:56.106 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:56.122+0000 7f083e5b5640 10 monclient: monmap: 2024-03-29T10:48:56.106 INFO:teuthology.misc.health.smithi088.stderr:epoch 0 2024-03-29T10:48:56.106 INFO:teuthology.misc.health.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:48:56.106 INFO:teuthology.misc.health.smithi088.stderr:last_changed 0.000000 2024-03-29T10:48:56.106 INFO:teuthology.misc.health.smithi088.stderr:created 0.000000 2024-03-29T10:48:56.106 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 0 (unknown) 2024-03-29T10:48:56.106 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:48:56.106 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-03-29T10:48:56.106 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-29T10:48:56.106 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-c 2024-03-29T10:48:56.106 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:48:56.106 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:56.123+0000 7f083e5b5640 1 librados: starting msgr at 2024-03-29T10:48:56.106 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:56.123+0000 7f083e5b5640 1 librados: starting objecter 2024-03-29T10:48:56.106 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:56.123+0000 7f083e5b5640 1 Processor -- start 2024-03-29T10:48:56.108 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:56.124+0000 7f083e5b5640 1 -- start start 2024-03-29T10:48:56.108 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:56.124+0000 7f083e5b5640 1 librados: setting wanted keys 2024-03-29T10:48:56.108 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:56.124+0000 7f083e5b5640 1 librados: calling monclient init 2024-03-29T10:48:56.108 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:56.124+0000 7f083e5b5640 10 monclient: init 2024-03-29T10:48:56.108 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:56.124+0000 7f083e5b5640 10 monclient: _reopen_session rank -1 2024-03-29T10:48:56.108 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:56.124+0000 7f083e5b5640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:48:56.108 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:56.124+0000 7f083e5b5640 10 monclient(hunting): picked mon.noname-b con 0x7f08380a9d70 addr v1:172.21.15.196:6789/0 2024-03-29T10:48:56.109 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:56.128+0000 7f083e5b5640 10 monclient(hunting): picked mon.noname-c con 0x7f083813f650 addr v1:172.21.15.88:6790/0 2024-03-29T10:48:56.111 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:56.129+0000 7f0837fff640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f08380a9d70 0x7f08381d7020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.88:37901/0 (socket says 172.21.15.88:37901) 2024-03-29T10:48:56.111 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:56.129+0000 7f0837fff640 1 -- 172.21.15.88:0/3296016935 learned_addr learned my addr 172.21.15.88:0/3296016935 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-29T10:48:56.111 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:56.129+0000 7f083e5b5640 10 monclient(hunting): picked mon.noname-a con 0x7f08381457f0 addr v1:172.21.15.88:6789/0 2024-03-29T10:48:56.111 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:56.129+0000 7f083e5b5640 1 -- 172.21.15.88:0/3296016935 --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0838142cf0 con 0x7f08381457f0 2024-03-29T10:48:56.111 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:56.129+0000 7f083e5b5640 1 -- 172.21.15.88:0/3296016935 --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0838142390 con 0x7f08380a9d70 2024-03-29T10:48:56.111 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:56.129+0000 7f083e5b5640 1 -- 172.21.15.88:0/3296016935 --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08381428b0 con 0x7f083813f650 2024-03-29T10:48:56.111 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:56.129+0000 7f083e5b5640 10 monclient(hunting): _renew_subs 2024-03-29T10:48:56.111 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:56.129+0000 7f083e5b5640 10 monclient(hunting): authenticate will time out at 1258.365332s 2024-03-29T10:48:56.112 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:56.129+0000 7f0834ff9640 10 client.?.objecter ms_handle_connect 0x7f08380a9d70 2024-03-29T10:48:56.112 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:56.129+0000 7f0834ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:48:56.113 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:56.131+0000 7f0834ff9640 1 -- 172.21.15.88:0/3296016935 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3767011357 0 0) 0x7f0838142390 con 0x7f08380a9d70 2024-03-29T10:48:56.114 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:56.131+0000 7f0834ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:48:56.114 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:56.131+0000 7f0834ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:48:56.114 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:56.131+0000 7f0834ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:48:56.114 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:56.131+0000 7f0834ff9640 10 monclient(hunting): my global_id is 4510 2024-03-29T10:48:56.114 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:56.131+0000 7f0834ff9640 1 -- 172.21.15.88:0/3296016935 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0824004eb0 con 0x7f08380a9d70 2024-03-29T10:48:56.115 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:56.133+0000 7f0834ff9640 1 -- 172.21.15.88:0/3296016935 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3364942035 0 0) 0x7f0824004eb0 con 0x7f08380a9d70 2024-03-29T10:48:56.115 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:56.133+0000 7f0834ff9640 1 -- 172.21.15.88:0/3296016935 --> v1:172.21.15.196:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0838142390 con 0x7f08380a9d70 2024-03-29T10:48:56.115 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:56.133+0000 7f0834ff9640 1 -- 172.21.15.88:0/3296016935 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f0830001f80 con 0x7f08380a9d70 2024-03-29T10:48:56.115 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:56.133+0000 7f0834ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:56.115 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:56.133+0000 7f0834ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:48:56.115 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:56.133+0000 7f0834ff9640 10 monclient(hunting): dump: 2024-03-29T10:48:56.115 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:48:56.115 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:48:56.115 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:48:56.115 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:48:56.115 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:56.115 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:48:56.116 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:48:56.116 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:48:56.116 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:48:56.116 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:48:56.116 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:56.133+0000 7f0834ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:48:56.119 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:56.133+0000 7f0834ff9640 1 -- 172.21.15.88:0/3296016935 <== mon.1 v1:172.21.15.196:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3536248907 0 0) 0x7f0838142390 con 0x7f08380a9d70 2024-03-29T10:48:56.119 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:56.133+0000 7f0834ff9640 1 -- 172.21.15.88:0/3296016935 >> v1:172.21.15.88:6790/0 conn(0x7f083813f650 legacy=0x7f08380b0ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:56.119 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:56.133+0000 7f0834ff9640 1 -- 172.21.15.88:0/3296016935 >> v1:172.21.15.88:6789/0 conn(0x7f08381457f0 legacy=0x7f08381e2940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:56.119 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:56.133+0000 7f0834ff9640 10 monclient: _finish_hunting 0 2024-03-29T10:48:56.119 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:56.134+0000 7f0834ff9640 1 monclient: found mon.b 2024-03-29T10:48:56.119 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:56.134+0000 7f0834ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:48:56.119 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:56.134+0000 7f0834ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T10:48:56.119 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:56.134+0000 7f0834ff9640 1 -- 172.21.15.88:0/3296016935 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0838142a40 con 0x7f08380a9d70 2024-03-29T10:48:56.119 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:56.134+0000 7f0834ff9640 10 monclient: _finish_auth 0 2024-03-29T10:48:56.119 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:56.134+0000 7f0834ff9640 10 monclient: _check_auth_tickets 2024-03-29T10:48:56.120 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:56.134+0000 7f0834ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:48:56.120 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:56.134+0000 7f0834ff9640 10 client.?.objecter ms_handle_connect 0x7f08381457f0 2024-03-29T10:48:56.120 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:56.134+0000 7f083e5b5640 5 monclient: authenticate success, global_id 4510 2024-03-29T10:48:56.120 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:56.134+0000 7f083e5b5640 10 monclient: _renew_subs 2024-03-29T10:48:56.120 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:56.134+0000 7f083e5b5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T10:48:56.120 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:56.134+0000 7f083e5b5640 1 -- 172.21.15.88:0/3296016935 --> v1:172.21.15.196:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08381d7790 con 0x7f08380a9d70 2024-03-29T10:48:56.120 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:56.135+0000 7f083e5b5640 10 client.4510.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:48:56.120 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:56.135+0000 7f083e5b5640 10 monclient: _renew_subs 2024-03-29T10:48:56.120 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:56.135+0000 7f083e5b5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T10:48:56.120 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:56.135+0000 7f083e5b5640 1 -- 172.21.15.88:0/3296016935 --> v1:172.21.15.196:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0838141a40 con 0x7f08380a9d70 2024-03-29T10:48:56.120 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:56.135+0000 7f083e5b5640 1 librados: init done 2024-03-29T10:48:56.122 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:56.135+0000 7f0834ff9640 10 client.4510.objecter resend_mon_ops 2024-03-29T10:48:56.122 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:56.135+0000 7f0834ff9640 1 -- 172.21.15.88:0/3296016935 <== mon.1 v1:172.21.15.196:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0830004e10 con 0x7f08380a9d70 2024-03-29T10:48:56.122 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:56.135+0000 7f0834ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:48:56.122 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:56.135+0000 7f0834ff9640 1 -- 172.21.15.88:0/3296016935 <== mon.1 v1:172.21.15.196:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f0830005260 con 0x7f08380a9d70 2024-03-29T10:48:56.122 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:56.135+0000 7f0834ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:56.122 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:56.135+0000 7f0834ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:48:56.122 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:56.135+0000 7f0834ff9640 10 monclient: dump: 2024-03-29T10:48:56.122 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:48:56.122 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:48:56.122 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:48:56.122 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:48:56.122 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:56.123 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:48:56.123 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:48:56.123 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:48:56.123 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:48:56.123 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:48:56.123 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:56.136+0000 7f0835ffb640 4 set_mon_vals no callback set 2024-03-29T10:48:56.124 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:56.141+0000 7f0834ff9640 1 -- 172.21.15.88:0/3296016935 <== mon.1 v1:172.21.15.196:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4235629220 0 0) 0x7f08300053c0 con 0x7f08380a9d70 2024-03-29T10:48:56.124 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:56.141+0000 7f083e5b5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:48:56.124 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:56.141+0000 7f083e5b5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:48:56.125 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:56.141+0000 7f083e5b5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T10:48:56.125 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:56.142+0000 7f083e5b5640 1 -- 172.21.15.88:0/3296016935 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0818005080 con 0x7f08380a9d70 2024-03-29T10:48:56.125 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:56.142+0000 7f0834ff9640 1 -- 172.21.15.88:0/3296016935 <== mon.1 v1:172.21.15.196:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4257673582 0 0) 0x7f0830084fc0 con 0x7f08380a9d70 2024-03-29T10:48:56.125 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:56.142+0000 7f0834ff9640 10 client.4510.objecter ms_dispatch 0x7f0838000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:48:56.125 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:56.142+0000 7f0834ff9640 3 client.4510.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:48:56.125 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:56.142+0000 7f0834ff9640 3 client.4510.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:48:56.125 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:56.142+0000 7f0834ff9640 20 client.4510.objecter dump_active .. 0 homeless 2024-03-29T10:48:56.125 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:56.142+0000 7f08377fe640 10 monclient: get_auth_request con 0x7f082406dc60 auth_method 0 2024-03-29T10:48:56.125 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:56.143+0000 7f0834ff9640 10 client.4510.objecter ms_handle_connect 0x7f082406dc60 2024-03-29T10:48:56.134 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:56.152+0000 7f0834ff9640 1 -- 172.21.15.88:0/3296016935 <== mon.1 v1:172.21.15.196:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f08300508a0 con 0x7f08380a9d70 2024-03-29T10:48:56.134 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:56.152+0000 7f0834ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:48:56.134 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:56.152+0000 7f0834ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:48:56.468 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:56.486+0000 7f083e5b5640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:48:56.468 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:56.486+0000 7f083e5b5640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:48:56.468 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:56.486+0000 7f083e5b5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T10:48:56.468 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:56.486+0000 7f083e5b5640 1 -- 172.21.15.88:0/3296016935 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0818005330 con 0x7f08380a9d70 2024-03-29T10:48:56.472 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:56.490+0000 7f0834ff9640 1 -- 172.21.15.88:0/3296016935 <== mon.1 v1:172.21.15.196:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 344506374) 0x7f08300527d0 con 0x7f08380a9d70 2024-03-29T10:48:56.472 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:56.490+0000 7f0834ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:48:56.472 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:56.490+0000 7f0834ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:48:56.472 INFO:teuthology.misc.health.smithi088.stdout: 2024-03-29T10:48:56.473 INFO:teuthology.misc.health.smithi088.stdout:{"epoch":10,"fsid":"869f5188-f92b-42dd-a46d-8c646e0388a9","created":"2024-03-29T10:40:40.285087+0000","modified":"2024-03-29T10:40:42.536230+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:40:42.536230+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a26c0c1c-8de0-438c-a3f6-3df10ea61574","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5de24915-8dec-49b6-8b16-f3170197c591","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ac054af3-d589-4565-9644-e6af75975c53","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3ec4f045-c714-4a22-bd1a-e07330b45ab3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"cddb9867-8e30-4951-9b13-3d1e78b9c45a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dc6d9911-785d-432f-8b53-56a1ccab302f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"77c88437-b587-496e-8b4d-1824ea8b5305","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ce963d96-5a8e-44c3-99de-c0dbebe234ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:48:56.478 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:56.493+0000 7f083e5b5640 10 librados: watch_flush enter 2024-03-29T10:48:56.479 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:56.493+0000 7f083e5b5640 10 librados: watch_flush exit 2024-03-29T10:48:56.479 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:56.493+0000 7f083e5b5640 20 client.4510.objecter shutdown clearing up homeless session... 2024-03-29T10:48:56.479 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:56.493+0000 7f083e5b5640 10 client.4510.objecter successfully canceled tick 2024-03-29T10:48:56.479 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:56.496+0000 7f083e5b5640 1 -- 172.21.15.88:0/3296016935 >> v1:172.21.15.88:6816/1752861349 conn(0x7f082406dc60 legacy=0x7f0824070120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:56.479 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:56.496+0000 7f083e5b5640 10 monclient: shutdown 2024-03-29T10:48:56.479 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:56.496+0000 7f083e5b5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:48:56.479 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:56.496+0000 7f083e5b5640 1 -- 172.21.15.88:0/3296016935 >> v1:172.21.15.196:6789/0 conn(0x7f08380a9d70 legacy=0x7f08381d7020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:56.479 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:56.496+0000 7f083e5b5640 1 -- 172.21.15.88:0/3296016935 shutdown_connections 2024-03-29T10:48:56.479 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:56.496+0000 7f083e5b5640 1 -- 172.21.15.88:0/3296016935 >> 172.21.15.88:0/3296016935 conn(0x7f0838066a00 msgr2=0x7f083813a770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:48:56.480 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:56.497+0000 7f083e5b5640 1 -- 172.21.15.88:0/3296016935 shutdown_connections 2024-03-29T10:48:56.480 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:56.497+0000 7f083e5b5640 1 -- 172.21.15.88:0/3296016935 wait complete. 2024-03-29T10:48:56.480 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:48:56.497+0000 7f083e5b5640 1 librados: shutdown 2024-03-29T10:48:56.496 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:49:03.304 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:49:03.455 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:03.473+0000 7f9a69123640 10 monclient: get_monmap_and_config 2024-03-29T10:49:03.456 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:03.473+0000 7f9a69123640 10 monclient: build_initial_monmap 2024-03-29T10:49:03.456 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:03.473+0000 7f9a69123640 10 monclient: monmap: 2024-03-29T10:49:03.456 INFO:teuthology.misc.health.smithi088.stderr:epoch 0 2024-03-29T10:49:03.456 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:49:03.456 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:49:03.474481+0000 2024-03-29T10:49:03.456 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:49:03.474481+0000 2024-03-29T10:49:03.456 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 0 (unknown) 2024-03-29T10:49:03.456 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:49:03.456 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-03-29T10:49:03.456 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-29T10:49:03.456 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-b 2024-03-29T10:49:03.456 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:49:03.459 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:03.477+0000 7f9a69123640 1 Processor -- start 2024-03-29T10:49:03.460 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:03.477+0000 7f9a69123640 1 -- start start 2024-03-29T10:49:03.460 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:03.478+0000 7f9a69123640 10 monclient: init 2024-03-29T10:49:03.460 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:03.478+0000 7f9a69123640 10 monclient: _reopen_session rank -1 2024-03-29T10:49:03.460 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:03.478+0000 7f9a69123640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:49:03.460 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:03.478+0000 7f9a69123640 10 monclient(hunting): picked mon.noname-c con 0x7f9a64071de0 addr v1:172.21.15.196:6789/0 2024-03-29T10:49:03.461 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:03.478+0000 7f9a69123640 10 monclient(hunting): picked mon.noname-a con 0x7f9a640721c0 addr v1:172.21.15.88:6789/0 2024-03-29T10:49:03.461 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:03.478+0000 7f9a69123640 10 monclient(hunting): picked mon.noname-b con 0x7f9a64145ae0 addr v1:172.21.15.88:6790/0 2024-03-29T10:49:03.461 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:03.478+0000 7f9a69123640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a64145650 con 0x7f9a640721c0 2024-03-29T10:49:03.461 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:03.478+0000 7f9a69123640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a640ae260 con 0x7f9a64071de0 2024-03-29T10:49:03.461 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:03.478+0000 7f9a69123640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a641420d0 con 0x7f9a64145ae0 2024-03-29T10:49:03.461 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:03.478+0000 7f9a69123640 10 monclient(hunting): _renew_subs 2024-03-29T10:49:03.461 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:03.478+0000 7f9a69123640 10 monclient(hunting): authenticate will time out at 1265.714885s 2024-03-29T10:49:03.462 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:03.479+0000 7f9a6259c640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f9a640721c0 0x7f9a6413f400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:46240/0 (socket says 172.21.15.88:46240) 2024-03-29T10:49:03.462 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:03.479+0000 7f9a6259c640 1 -- 172.21.15.88:0/3537671165 learned_addr learned my addr 172.21.15.88:0/3537671165 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-29T10:49:03.466 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:03.482+0000 7f9a61d9b640 1 -- 172.21.15.88:0/3537671165 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4031009088 0 0) 0x7f9a641420d0 con 0x7f9a64145ae0 2024-03-29T10:49:03.466 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:03.483+0000 7f9a61d9b640 10 monclient(hunting): _negotiate 2024-03-29T10:49:03.466 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:03.483+0000 7f9a61d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:03.466 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:03.483+0000 7f9a61d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:03.466 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:03.483+0000 7f9a61d9b640 10 monclient(hunting): my global_id is 4550 2024-03-29T10:49:03.467 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:03.483+0000 7f9a61d9b640 1 -- 172.21.15.88:0/3537671165 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9a4c003520 con 0x7f9a64145ae0 2024-03-29T10:49:03.467 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:03.483+0000 7f9a61d9b640 1 -- 172.21.15.88:0/3537671165 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2523809051 0 0) 0x7f9a64145650 con 0x7f9a640721c0 2024-03-29T10:49:03.471 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:03.483+0000 7f9a61d9b640 10 monclient(hunting): _negotiate 2024-03-29T10:49:03.472 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:03.483+0000 7f9a61d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:03.472 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:03.483+0000 7f9a61d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:03.472 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:03.483+0000 7f9a61d9b640 10 monclient(hunting): my global_id is 4527 2024-03-29T10:49:03.472 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:03.483+0000 7f9a61d9b640 1 -- 172.21.15.88:0/3537671165 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9a641420d0 con 0x7f9a640721c0 2024-03-29T10:49:03.472 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:03.483+0000 7f9a61d9b640 1 -- 172.21.15.88:0/3537671165 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2197140621 0 0) 0x7f9a4c003520 con 0x7f9a64145ae0 2024-03-29T10:49:03.472 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:03.483+0000 7f9a61d9b640 1 -- 172.21.15.88:0/3537671165 >> v1:172.21.15.196:6789/0 conn(0x7f9a64071de0 legacy=0x7f9a640aed80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:03.472 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:03.483+0000 7f9a61d9b640 1 -- 172.21.15.88:0/3537671165 >> v1:172.21.15.88:6789/0 conn(0x7f9a640721c0 legacy=0x7f9a6413f400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:03.472 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:03.483+0000 7f9a61d9b640 10 monclient: _finish_hunting 0 2024-03-29T10:49:03.472 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:03.483+0000 7f9a61d9b640 1 monclient: found mon.noname-b 2024-03-29T10:49:03.472 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:03.483+0000 7f9a61d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:49:03.472 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:03.483+0000 7f9a61d9b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.88:6790/0 2024-03-29T10:49:03.472 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:03.483+0000 7f9a61d9b640 1 -- 172.21.15.88:0/3537671165 --> v1:172.21.15.88:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a640b0590 con 0x7f9a64145ae0 2024-03-29T10:49:03.472 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:03.483+0000 7f9a61d9b640 10 monclient: _finish_auth 0 2024-03-29T10:49:03.473 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:03.483+0000 7f9a61d9b640 10 monclient: _check_auth_tickets 2024-03-29T10:49:03.473 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:03.483+0000 7f9a61d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:49:03.473 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:03.483+0000 7f9a61d9b640 1 -- 172.21.15.88:0/3537671165 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f9a5c001a10 con 0x7f9a64145ae0 2024-03-29T10:49:03.473 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:03.483+0000 7f9a69123640 5 monclient: authenticate success, global_id 4550 2024-03-29T10:49:03.473 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:03.483+0000 7f9a69123640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:49:03.473 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:03.483+0000 7f9a61d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:03.473 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:03.483+0000 7f9a61d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:49:03.473 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:03.483+0000 7f9a61d9b640 10 monclient: dump: 2024-03-29T10:49:03.473 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:49:03.473 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:49:03.473 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:49:03.473 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:49:03.473 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:03.473 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:49:03.474 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:49:03.474 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:49:03.474 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:49:03.474 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:49:03.474 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:03.483+0000 7f9a69123640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:49:03.474 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:03.485+0000 7f9a61d9b640 1 -- 172.21.15.88:0/3537671165 <== mon.2 v1:172.21.15.88:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9a5c001b70 con 0x7f9a64145ae0 2024-03-29T10:49:03.474 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:03.485+0000 7f9a61d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:49:03.474 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:03.485+0000 7f9a61d9b640 1 -- 172.21.15.88:0/3537671165 <== mon.2 v1:172.21.15.88:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f9a5c001f00 con 0x7f9a64145ae0 2024-03-29T10:49:03.474 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:03.485+0000 7f9a61d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:03.474 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:03.485+0000 7f9a61d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:49:03.474 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:03.485+0000 7f9a61d9b640 10 monclient: dump: 2024-03-29T10:49:03.474 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:49:03.474 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:49:03.474 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:49:03.475 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:49:03.475 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:03.475 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:49:03.475 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:49:03.475 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:49:03.475 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:49:03.475 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:49:03.475 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:03.485+0000 7f9a69123640 10 monclient: get_monmap_and_config success 2024-03-29T10:49:03.475 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:03.485+0000 7f9a69123640 4 set_mon_vals no callback set 2024-03-29T10:49:03.475 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:03.486+0000 7f9a69123640 10 monclient: shutdown 2024-03-29T10:49:03.475 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:03.486+0000 7f9a69123640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:49:03.475 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:03.486+0000 7f9a69123640 1 -- 172.21.15.88:0/3537671165 >> v1:172.21.15.88:6790/0 conn(0x7f9a64145ae0 legacy=0x7f9a64145f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:03.475 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:03.487+0000 7f9a69123640 1 -- 172.21.15.88:0/3537671165 shutdown_connections 2024-03-29T10:49:03.475 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:03.487+0000 7f9a69123640 1 -- 172.21.15.88:0/3537671165 >> 172.21.15.88:0/3537671165 conn(0x7f9a6406b660 msgr2=0x7f9a6406ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:49:03.476 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:03.487+0000 7f9a69123640 1 -- 172.21.15.88:0/3537671165 shutdown_connections 2024-03-29T10:49:03.476 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:03.487+0000 7f9a69123640 1 -- 172.21.15.88:0/3537671165 wait complete. 2024-03-29T10:49:03.476 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:03.488+0000 7f9a69123640 10 monclient: build_initial_monmap 2024-03-29T10:49:03.476 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:03.488+0000 7f9a69123640 10 monclient: monmap: 2024-03-29T10:49:03.476 INFO:teuthology.misc.health.smithi088.stderr:epoch 0 2024-03-29T10:49:03.476 INFO:teuthology.misc.health.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:49:03.476 INFO:teuthology.misc.health.smithi088.stderr:last_changed 0.000000 2024-03-29T10:49:03.476 INFO:teuthology.misc.health.smithi088.stderr:created 0.000000 2024-03-29T10:49:03.476 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 0 (unknown) 2024-03-29T10:49:03.476 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:49:03.476 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-03-29T10:49:03.476 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-29T10:49:03.476 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-c 2024-03-29T10:49:03.476 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:49:03.477 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:03.488+0000 7f9a69123640 1 librados: starting msgr at 2024-03-29T10:49:03.477 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:03.488+0000 7f9a69123640 1 librados: starting objecter 2024-03-29T10:49:03.477 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:03.489+0000 7f9a69123640 1 Processor -- start 2024-03-29T10:49:03.477 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:03.490+0000 7f9a69123640 1 -- start start 2024-03-29T10:49:03.478 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:03.493+0000 7f9a69123640 1 librados: setting wanted keys 2024-03-29T10:49:03.478 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:03.493+0000 7f9a69123640 1 librados: calling monclient init 2024-03-29T10:49:03.478 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:03.493+0000 7f9a69123640 10 monclient: init 2024-03-29T10:49:03.478 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:03.494+0000 7f9a69123640 10 monclient: _reopen_session rank -1 2024-03-29T10:49:03.479 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:03.494+0000 7f9a69123640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:49:03.479 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:03.494+0000 7f9a69123640 10 monclient(hunting): picked mon.noname-b con 0x7f9a64071de0 addr v1:172.21.15.196:6789/0 2024-03-29T10:49:03.479 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:03.494+0000 7f9a69123640 10 monclient(hunting): picked mon.noname-a con 0x7f9a64145ae0 addr v1:172.21.15.88:6789/0 2024-03-29T10:49:03.481 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:03.495+0000 7f9a69123640 10 monclient(hunting): picked mon.noname-c con 0x7f9a641d3c20 addr v1:172.21.15.88:6790/0 2024-03-29T10:49:03.481 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:03.495+0000 7f9a69123640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a64139190 con 0x7f9a64145ae0 2024-03-29T10:49:03.482 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:03.495+0000 7f9a69123640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a641d7330 con 0x7f9a64071de0 2024-03-29T10:49:03.482 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:03.495+0000 7f9a69123640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a641d7f60 con 0x7f9a641d3c20 2024-03-29T10:49:03.482 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:03.495+0000 7f9a69123640 10 monclient(hunting): _renew_subs 2024-03-29T10:49:03.482 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:03.495+0000 7f9a69123640 10 monclient(hunting): authenticate will time out at 1265.731264s 2024-03-29T10:49:03.482 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:03.495+0000 7f9a6259c640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f9a64145ae0 0x7f9a641377d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:57840/0 (socket says 172.21.15.88:57840) 2024-03-29T10:49:03.482 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:03.495+0000 7f9a6259c640 1 -- 172.21.15.88:0/2015827687 learned_addr learned my addr 172.21.15.88:0/2015827687 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-29T10:49:03.482 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:03.496+0000 7f9a4b7fe640 10 client.?.objecter ms_handle_connect 0x7f9a64145ae0 2024-03-29T10:49:03.482 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:03.496+0000 7f9a4b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:49:03.482 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:03.497+0000 7f9a4b7fe640 10 client.?.objecter ms_handle_connect 0x7f9a641d3c20 2024-03-29T10:49:03.482 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:03.497+0000 7f9a4b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:49:03.482 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:03.498+0000 7f9a4b7fe640 1 -- 172.21.15.88:0/2015827687 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3054472168 0 0) 0x7f9a64139190 con 0x7f9a64145ae0 2024-03-29T10:49:03.483 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:03.498+0000 7f9a4b7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:49:03.483 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:03.498+0000 7f9a4b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:03.483 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:03.498+0000 7f9a4b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:03.483 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:03.498+0000 7f9a4b7fe640 10 monclient(hunting): my global_id is 4530 2024-03-29T10:49:03.483 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:03.498+0000 7f9a4b7fe640 1 -- 172.21.15.88:0/2015827687 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9a50004eb0 con 0x7f9a64145ae0 2024-03-29T10:49:03.483 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:03.498+0000 7f9a4b7fe640 1 -- 172.21.15.88:0/2015827687 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2408512374 0 0) 0x7f9a641d7f60 con 0x7f9a641d3c20 2024-03-29T10:49:03.483 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:03.498+0000 7f9a4b7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:49:03.483 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:03.498+0000 7f9a4b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:03.483 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:03.498+0000 7f9a4b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:03.483 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:03.498+0000 7f9a4b7fe640 10 monclient(hunting): my global_id is 4553 2024-03-29T10:49:03.484 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:03.498+0000 7f9a4b7fe640 1 -- 172.21.15.88:0/2015827687 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9a64139190 con 0x7f9a641d3c20 2024-03-29T10:49:03.484 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:03.498+0000 7f9a4b7fe640 1 -- 172.21.15.88:0/2015827687 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4222601313 0 0) 0x7f9a50004eb0 con 0x7f9a64145ae0 2024-03-29T10:49:03.484 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:03.498+0000 7f9a4b7fe640 1 -- 172.21.15.88:0/2015827687 --> v1:172.21.15.88:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9a641d7f60 con 0x7f9a64145ae0 2024-03-29T10:49:03.484 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:03.498+0000 7f9a4b7fe640 1 -- 172.21.15.88:0/2015827687 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f9a580019f0 con 0x7f9a64145ae0 2024-03-29T10:49:03.484 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:03.498+0000 7f9a4b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:03.484 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:03.498+0000 7f9a4b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:49:03.484 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:03.498+0000 7f9a4b7fe640 10 monclient(hunting): dump: 2024-03-29T10:49:03.484 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:49:03.484 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:49:03.484 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:49:03.484 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:49:03.484 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:03.484 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:49:03.484 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:49:03.485 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:49:03.485 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:49:03.485 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:49:03.485 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:03.498+0000 7f9a4b7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:49:03.489 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:03.500+0000 7f9a4b7fe640 10 client.?.objecter ms_handle_connect 0x7f9a64071de0 2024-03-29T10:49:03.489 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:03.500+0000 7f9a4b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:49:03.489 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:03.500+0000 7f9a4b7fe640 1 -- 172.21.15.88:0/2015827687 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2947867197 0 0) 0x7f9a64139190 con 0x7f9a641d3c20 2024-03-29T10:49:03.489 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:03.500+0000 7f9a4b7fe640 1 -- 172.21.15.88:0/2015827687 --> v1:172.21.15.88:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9a50004eb0 con 0x7f9a641d3c20 2024-03-29T10:49:03.490 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:03.500+0000 7f9a4b7fe640 1 -- 172.21.15.88:0/2015827687 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3955875577 0 0) 0x7f9a641d7330 con 0x7f9a64071de0 2024-03-29T10:49:03.490 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:03.500+0000 7f9a4b7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:49:03.490 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:03.500+0000 7f9a4b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:03.490 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:03.500+0000 7f9a4b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:03.490 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:03.500+0000 7f9a4b7fe640 10 monclient(hunting): my global_id is 4513 2024-03-29T10:49:03.490 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:03.500+0000 7f9a4b7fe640 1 -- 172.21.15.88:0/2015827687 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9a64139190 con 0x7f9a64071de0 2024-03-29T10:49:03.490 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:03.500+0000 7f9a4b7fe640 1 -- 172.21.15.88:0/2015827687 <== mon.0 v1:172.21.15.88:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1110832393 0 0) 0x7f9a641d7f60 con 0x7f9a64145ae0 2024-03-29T10:49:03.490 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:03.500+0000 7f9a4b7fe640 1 -- 172.21.15.88:0/2015827687 >> v1:172.21.15.88:6790/0 conn(0x7f9a641d3c20 legacy=0x7f9a641d4330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:03.490 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:03.500+0000 7f9a4b7fe640 1 -- 172.21.15.88:0/2015827687 >> v1:172.21.15.196:6789/0 conn(0x7f9a64071de0 legacy=0x7f9a641d7850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:03.490 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:03.500+0000 7f9a4b7fe640 10 monclient: _finish_hunting 0 2024-03-29T10:49:03.490 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:03.500+0000 7f9a4b7fe640 1 monclient: found mon.a 2024-03-29T10:49:03.490 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:03.500+0000 7f9a4b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:49:03.490 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:03.500+0000 7f9a4b7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-03-29T10:49:03.491 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:03.500+0000 7f9a4b7fe640 1 -- 172.21.15.88:0/2015827687 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a641d80f0 con 0x7f9a64145ae0 2024-03-29T10:49:03.491 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:03.500+0000 7f9a4b7fe640 10 monclient: _finish_auth 0 2024-03-29T10:49:03.491 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:03.500+0000 7f9a4b7fe640 10 monclient: _check_auth_tickets 2024-03-29T10:49:03.491 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:03.500+0000 7f9a4b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:49:03.491 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:03.501+0000 7f9a4b7fe640 1 -- 172.21.15.88:0/2015827687 <== mon.0 v1:172.21.15.88:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9a580047e0 con 0x7f9a64145ae0 2024-03-29T10:49:03.491 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:03.501+0000 7f9a4b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:49:03.491 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:03.501+0000 7f9a4b7fe640 1 -- 172.21.15.88:0/2015827687 <== mon.0 v1:172.21.15.88:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f9a58004af0 con 0x7f9a64145ae0 2024-03-29T10:49:03.491 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:03.501+0000 7f9a4b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:03.491 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:03.501+0000 7f9a4b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:49:03.491 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:03.501+0000 7f9a4b7fe640 10 monclient: dump: 2024-03-29T10:49:03.492 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:49:03.492 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:49:03.492 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:49:03.492 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:49:03.492 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:03.492 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:49:03.492 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:49:03.492 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:49:03.492 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:49:03.492 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:49:03.492 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:03.501+0000 7f9a69123640 5 monclient: authenticate success, global_id 4530 2024-03-29T10:49:03.492 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:03.501+0000 7f9a69123640 10 monclient: _renew_subs 2024-03-29T10:49:03.492 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:03.501+0000 7f9a69123640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-03-29T10:49:03.492 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:03.501+0000 7f9a69123640 1 -- 172.21.15.88:0/2015827687 --> v1:172.21.15.88:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a641d4a40 con 0x7f9a64145ae0 2024-03-29T10:49:03.493 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:03.501+0000 7f9a69123640 10 client.4530.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:49:03.493 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:03.501+0000 7f9a69123640 10 monclient: _renew_subs 2024-03-29T10:49:03.493 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:03.501+0000 7f9a69123640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-03-29T10:49:03.493 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:03.501+0000 7f9a69123640 1 -- 172.21.15.88:0/2015827687 --> v1:172.21.15.88:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a641380c0 con 0x7f9a64145ae0 2024-03-29T10:49:03.493 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:03.501+0000 7f9a69123640 1 librados: init done 2024-03-29T10:49:03.493 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:03.502+0000 7f9a6159a640 4 set_mon_vals no callback set 2024-03-29T10:49:03.493 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:03.503+0000 7f9a69123640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:49:03.493 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:03.503+0000 7f9a69123640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:49:03.493 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:03.503+0000 7f9a69123640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-03-29T10:49:03.493 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:03.503+0000 7f9a69123640 1 -- 172.21.15.88:0/2015827687 --> v1:172.21.15.88:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a44005080 con 0x7f9a64145ae0 2024-03-29T10:49:03.493 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:03.504+0000 7f9a4b7fe640 1 -- 172.21.15.88:0/2015827687 <== mon.0 v1:172.21.15.88:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4235629220 0 0) 0x7f9a5801aac0 con 0x7f9a64145ae0 2024-03-29T10:49:03.494 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:03.505+0000 7f9a4b7fe640 1 -- 172.21.15.88:0/2015827687 <== mon.0 v1:172.21.15.88:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4257673582 0 0) 0x7f9a58084a90 con 0x7f9a64145ae0 2024-03-29T10:49:03.494 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:03.505+0000 7f9a4b7fe640 10 client.4530.objecter ms_dispatch 0x7f9a64000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:49:03.494 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:03.505+0000 7f9a4b7fe640 3 client.4530.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:49:03.494 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:03.505+0000 7f9a4b7fe640 3 client.4530.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:49:03.494 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:03.505+0000 7f9a4b7fe640 20 client.4530.objecter dump_active .. 0 homeless 2024-03-29T10:49:03.494 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:03.505+0000 7f9a62d9d640 10 monclient: get_auth_request con 0x7f9a5006de30 auth_method 0 2024-03-29T10:49:03.494 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:03.505+0000 7f9a4b7fe640 10 client.4530.objecter ms_handle_connect 0x7f9a5006de30 2024-03-29T10:49:03.496 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:03.513+0000 7f9a4b7fe640 1 -- 172.21.15.88:0/2015827687 <== mon.0 v1:172.21.15.88:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9a580501a0 con 0x7f9a64145ae0 2024-03-29T10:49:03.496 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:03.513+0000 7f9a4b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:49:03.496 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:03.513+0000 7f9a4b7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:49:03.816 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:03.834+0000 7f9a69123640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:49:03.816 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:03.834+0000 7f9a69123640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:49:03.816 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:03.834+0000 7f9a69123640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-03-29T10:49:03.816 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:03.834+0000 7f9a69123640 1 -- 172.21.15.88:0/2015827687 --> v1:172.21.15.88:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9a44005330 con 0x7f9a64145ae0 2024-03-29T10:49:03.818 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:03.836+0000 7f9a4b7fe640 1 -- 172.21.15.88:0/2015827687 <== mon.0 v1:172.21.15.88:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 344506374) 0x7f9a580522a0 con 0x7f9a64145ae0 2024-03-29T10:49:03.818 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:03.836+0000 7f9a4b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:49:03.819 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:03.836+0000 7f9a4b7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:49:03.819 INFO:teuthology.misc.health.smithi088.stdout: 2024-03-29T10:49:03.819 INFO:teuthology.misc.health.smithi088.stdout:{"epoch":10,"fsid":"869f5188-f92b-42dd-a46d-8c646e0388a9","created":"2024-03-29T10:40:40.285087+0000","modified":"2024-03-29T10:40:42.536230+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:40:42.536230+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a26c0c1c-8de0-438c-a3f6-3df10ea61574","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5de24915-8dec-49b6-8b16-f3170197c591","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ac054af3-d589-4565-9644-e6af75975c53","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3ec4f045-c714-4a22-bd1a-e07330b45ab3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"cddb9867-8e30-4951-9b13-3d1e78b9c45a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dc6d9911-785d-432f-8b53-56a1ccab302f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"77c88437-b587-496e-8b4d-1824ea8b5305","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ce963d96-5a8e-44c3-99de-c0dbebe234ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:49:03.821 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:03.839+0000 7f9a69123640 10 librados: watch_flush enter 2024-03-29T10:49:03.821 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:03.839+0000 7f9a69123640 10 librados: watch_flush exit 2024-03-29T10:49:03.821 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:03.839+0000 7f9a69123640 20 client.4530.objecter shutdown clearing up homeless session... 2024-03-29T10:49:03.821 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:03.839+0000 7f9a69123640 10 client.4530.objecter successfully canceled tick 2024-03-29T10:49:03.822 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:03.839+0000 7f9a69123640 1 -- 172.21.15.88:0/2015827687 >> v1:172.21.15.88:6816/1752861349 conn(0x7f9a5006de30 legacy=0x7f9a500702f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:03.822 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:03.839+0000 7f9a69123640 10 monclient: shutdown 2024-03-29T10:49:03.822 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:03.839+0000 7f9a69123640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:49:03.822 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:03.839+0000 7f9a69123640 1 -- 172.21.15.88:0/2015827687 >> v1:172.21.15.88:6789/0 conn(0x7f9a64145ae0 legacy=0x7f9a641377d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:03.822 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:03.840+0000 7f9a69123640 1 -- 172.21.15.88:0/2015827687 shutdown_connections 2024-03-29T10:49:03.823 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:03.840+0000 7f9a69123640 1 -- 172.21.15.88:0/2015827687 >> 172.21.15.88:0/2015827687 conn(0x7f9a6406b660 msgr2=0x7f9a641304f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:49:03.824 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:03.842+0000 7f9a69123640 1 -- 172.21.15.88:0/2015827687 shutdown_connections 2024-03-29T10:49:03.824 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:03.842+0000 7f9a69123640 1 -- 172.21.15.88:0/2015827687 wait complete. 2024-03-29T10:49:03.824 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:03.842+0000 7f9a69123640 1 librados: shutdown 2024-03-29T10:49:03.842 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:49:10.648 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:49:10.796 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:10.813+0000 7fa7ced1c640 10 monclient: get_monmap_and_config 2024-03-29T10:49:10.796 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:10.813+0000 7fa7ced1c640 10 monclient: build_initial_monmap 2024-03-29T10:49:10.796 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:10.813+0000 7fa7ced1c640 10 monclient: monmap: 2024-03-29T10:49:10.796 INFO:teuthology.misc.health.smithi088.stderr:epoch 0 2024-03-29T10:49:10.796 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:49:10.796 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:49:10.814589+0000 2024-03-29T10:49:10.796 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:49:10.814589+0000 2024-03-29T10:49:10.796 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 0 (unknown) 2024-03-29T10:49:10.796 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:49:10.796 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-03-29T10:49:10.796 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-29T10:49:10.797 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-b 2024-03-29T10:49:10.797 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:49:10.797 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:10.814+0000 7fa7ced1c640 1 Processor -- start 2024-03-29T10:49:10.797 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:10.814+0000 7fa7ced1c640 1 -- start start 2024-03-29T10:49:10.798 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:10.814+0000 7fa7ced1c640 10 monclient: init 2024-03-29T10:49:10.798 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:10.815+0000 7fa7ced1c640 10 monclient: _reopen_session rank -1 2024-03-29T10:49:10.798 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:10.815+0000 7fa7ced1c640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:49:10.798 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:10.815+0000 7fa7ced1c640 10 monclient(hunting): picked mon.noname-a con 0x7fa7c80b0770 addr v1:172.21.15.88:6789/0 2024-03-29T10:49:10.798 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:10.815+0000 7fa7ced1c640 10 monclient(hunting): picked mon.noname-c con 0x7fa7c814a700 addr v1:172.21.15.196:6789/0 2024-03-29T10:49:10.798 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:10.815+0000 7fa7ced1c640 10 monclient(hunting): picked mon.noname-b con 0x7fa7c80ac5a0 addr v1:172.21.15.88:6790/0 2024-03-29T10:49:10.798 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:10.815+0000 7fa7ced1c640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7c80aba70 con 0x7fa7c80b0770 2024-03-29T10:49:10.798 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:10.815+0000 7fa7ced1c640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7c80aecb0 con 0x7fa7c814a700 2024-03-29T10:49:10.798 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:10.815+0000 7fa7ced1c640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7c80af150 con 0x7fa7c80ac5a0 2024-03-29T10:49:10.798 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:10.815+0000 7fa7ced1c640 10 monclient(hunting): _renew_subs 2024-03-29T10:49:10.798 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:10.815+0000 7fa7ced1c640 10 monclient(hunting): authenticate will time out at 1273.051522s 2024-03-29T10:49:10.798 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:10.815+0000 7fa7c7fff640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7fa7c814a700 0x7fa7c80abe90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.88:59780/0 (socket says 172.21.15.88:59780) 2024-03-29T10:49:10.799 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:10.815+0000 7fa7c7fff640 1 -- 172.21.15.88:0/1755628401 learned_addr learned my addr 172.21.15.88:0/1755628401 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-29T10:49:10.800 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:10.816+0000 7fa7c77fe640 1 -- 172.21.15.88:0/1755628401 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 674226709 0 0) 0x7fa7c80aecb0 con 0x7fa7c814a700 2024-03-29T10:49:10.800 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:10.816+0000 7fa7c77fe640 10 monclient(hunting): _negotiate 2024-03-29T10:49:10.800 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:10.816+0000 7fa7c77fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:10.800 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:10.816+0000 7fa7c77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:10.800 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:10.816+0000 7fa7c77fe640 10 monclient(hunting): my global_id is 4516 2024-03-29T10:49:10.800 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:10.816+0000 7fa7c77fe640 1 -- 172.21.15.88:0/1755628401 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa7b0003520 con 0x7fa7c814a700 2024-03-29T10:49:10.800 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:10.816+0000 7fa7c77fe640 1 -- 172.21.15.88:0/1755628401 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 57254370 0 0) 0x7fa7c80af150 con 0x7fa7c80ac5a0 2024-03-29T10:49:10.800 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:10.816+0000 7fa7c77fe640 10 monclient(hunting): _negotiate 2024-03-29T10:49:10.800 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:10.816+0000 7fa7c77fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:10.800 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:10.816+0000 7fa7c77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:10.801 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:10.816+0000 7fa7c77fe640 10 monclient(hunting): my global_id is 4556 2024-03-29T10:49:10.801 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:10.816+0000 7fa7c77fe640 1 -- 172.21.15.88:0/1755628401 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa7c80aecb0 con 0x7fa7c80ac5a0 2024-03-29T10:49:10.801 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:10.816+0000 7fa7c77fe640 1 -- 172.21.15.88:0/1755628401 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3077186823 0 0) 0x7fa7c80aba70 con 0x7fa7c80b0770 2024-03-29T10:49:10.801 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:10.816+0000 7fa7c77fe640 10 monclient(hunting): _negotiate 2024-03-29T10:49:10.801 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:10.816+0000 7fa7c77fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:10.801 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:10.816+0000 7fa7c77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:10.801 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:10.816+0000 7fa7c77fe640 10 monclient(hunting): my global_id is 4533 2024-03-29T10:49:10.801 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:10.816+0000 7fa7c77fe640 1 -- 172.21.15.88:0/1755628401 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa7c80af150 con 0x7fa7c80b0770 2024-03-29T10:49:10.801 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:10.817+0000 7fa7c77fe640 1 -- 172.21.15.88:0/1755628401 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1160845582 0 0) 0x7fa7b0003520 con 0x7fa7c814a700 2024-03-29T10:49:10.801 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:10.817+0000 7fa7c77fe640 1 -- 172.21.15.88:0/1755628401 >> v1:172.21.15.88:6790/0 conn(0x7fa7c80ac5a0 legacy=0x7fa7c8151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:10.801 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:10.817+0000 7fa7c77fe640 1 -- 172.21.15.88:0/1755628401 >> v1:172.21.15.88:6789/0 conn(0x7fa7c80b0770 legacy=0x7fa7c8149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:10.801 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:10.817+0000 7fa7c77fe640 10 monclient: _finish_hunting 0 2024-03-29T10:49:10.801 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:10.817+0000 7fa7c77fe640 1 monclient: found mon.noname-c 2024-03-29T10:49:10.802 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:10.817+0000 7fa7c77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:49:10.802 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:10.817+0000 7fa7c77fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.196:6789/0 2024-03-29T10:49:10.802 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:10.817+0000 7fa7c77fe640 1 -- 172.21.15.88:0/1755628401 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7c80ac9d0 con 0x7fa7c814a700 2024-03-29T10:49:10.802 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:10.817+0000 7fa7c77fe640 10 monclient: _finish_auth 0 2024-03-29T10:49:10.802 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:10.817+0000 7fa7c77fe640 10 monclient: _check_auth_tickets 2024-03-29T10:49:10.802 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:10.817+0000 7fa7c77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:49:10.802 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:10.817+0000 7fa7c77fe640 1 -- 172.21.15.88:0/1755628401 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7fa7bc0016b0 con 0x7fa7c814a700 2024-03-29T10:49:10.802 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:10.817+0000 7fa7c77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:10.802 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:10.817+0000 7fa7c77fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:49:10.802 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:10.817+0000 7fa7c77fe640 10 monclient: dump: 2024-03-29T10:49:10.802 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:49:10.802 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:49:10.802 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:49:10.802 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:49:10.803 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:10.803 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:49:10.803 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:49:10.803 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:49:10.803 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:49:10.803 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:49:10.803 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:10.817+0000 7fa7ced1c640 5 monclient: authenticate success, global_id 4516 2024-03-29T10:49:10.803 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:10.817+0000 7fa7ced1c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:49:10.803 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:10.817+0000 7fa7c77fe640 1 -- 172.21.15.88:0/1755628401 <== mon.1 v1:172.21.15.196:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa7bc001810 con 0x7fa7c814a700 2024-03-29T10:49:10.803 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:10.817+0000 7fa7c77fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:49:10.803 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:10.817+0000 7fa7c77fe640 1 -- 172.21.15.88:0/1755628401 <== mon.1 v1:172.21.15.196:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7fa7bc001c20 con 0x7fa7c814a700 2024-03-29T10:49:10.803 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:10.817+0000 7fa7c77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:10.803 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:10.817+0000 7fa7c77fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:49:10.803 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:10.817+0000 7fa7c77fe640 10 monclient: dump: 2024-03-29T10:49:10.804 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:49:10.804 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:49:10.804 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:49:10.804 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:49:10.804 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:10.804 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:49:10.804 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:49:10.804 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:49:10.804 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:49:10.804 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:49:10.804 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:10.817+0000 7fa7ced1c640 10 monclient: get_monmap_and_config success 2024-03-29T10:49:10.804 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:10.817+0000 7fa7ced1c640 4 set_mon_vals no callback set 2024-03-29T10:49:10.807 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:10.818+0000 7fa7ced1c640 10 monclient: shutdown 2024-03-29T10:49:10.807 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:10.818+0000 7fa7ced1c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:49:10.807 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:10.818+0000 7fa7ced1c640 1 -- 172.21.15.88:0/1755628401 >> v1:172.21.15.196:6789/0 conn(0x7fa7c814a700 legacy=0x7fa7c80abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:10.807 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:10.818+0000 7fa7ced1c640 1 -- 172.21.15.88:0/1755628401 shutdown_connections 2024-03-29T10:49:10.807 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:10.818+0000 7fa7ced1c640 1 -- 172.21.15.88:0/1755628401 >> 172.21.15.88:0/1755628401 conn(0x7fa7c806b660 msgr2=0x7fa7c806ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:49:10.807 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:10.819+0000 7fa7ced1c640 1 -- 172.21.15.88:0/1755628401 shutdown_connections 2024-03-29T10:49:10.808 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:10.819+0000 7fa7ced1c640 1 -- 172.21.15.88:0/1755628401 wait complete. 2024-03-29T10:49:10.808 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:10.820+0000 7fa7ced1c640 10 monclient: build_initial_monmap 2024-03-29T10:49:10.808 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:10.820+0000 7fa7ced1c640 10 monclient: monmap: 2024-03-29T10:49:10.808 INFO:teuthology.misc.health.smithi088.stderr:epoch 0 2024-03-29T10:49:10.808 INFO:teuthology.misc.health.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:49:10.808 INFO:teuthology.misc.health.smithi088.stderr:last_changed 0.000000 2024-03-29T10:49:10.808 INFO:teuthology.misc.health.smithi088.stderr:created 0.000000 2024-03-29T10:49:10.808 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 0 (unknown) 2024-03-29T10:49:10.808 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:49:10.808 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-03-29T10:49:10.808 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-29T10:49:10.808 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-c 2024-03-29T10:49:10.808 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:49:10.809 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:10.821+0000 7fa7ced1c640 1 librados: starting msgr at 2024-03-29T10:49:10.809 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:10.821+0000 7fa7ced1c640 1 librados: starting objecter 2024-03-29T10:49:10.809 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:10.821+0000 7fa7ced1c640 1 Processor -- start 2024-03-29T10:49:10.809 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:10.821+0000 7fa7ced1c640 1 -- start start 2024-03-29T10:49:10.809 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:10.821+0000 7fa7ced1c640 1 librados: setting wanted keys 2024-03-29T10:49:10.809 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:10.821+0000 7fa7ced1c640 1 librados: calling monclient init 2024-03-29T10:49:10.809 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:10.821+0000 7fa7ced1c640 10 monclient: init 2024-03-29T10:49:10.809 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:10.822+0000 7fa7ced1c640 10 monclient: _reopen_session rank -1 2024-03-29T10:49:10.809 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:10.822+0000 7fa7ced1c640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:49:10.809 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:10.823+0000 7fa7ced1c640 10 monclient(hunting): picked mon.noname-b con 0x7fa7c80ac5a0 addr v1:172.21.15.196:6789/0 2024-03-29T10:49:10.809 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:10.823+0000 7fa7ced1c640 10 monclient(hunting): picked mon.noname-c con 0x7fa7c80b0770 addr v1:172.21.15.88:6790/0 2024-03-29T10:49:10.809 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:10.823+0000 7fa7ced1c640 10 monclient(hunting): picked mon.noname-a con 0x7fa7c814a700 addr v1:172.21.15.88:6789/0 2024-03-29T10:49:10.809 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:10.823+0000 7fa7ced1c640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7c8127650 con 0x7fa7c814a700 2024-03-29T10:49:10.809 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:10.823+0000 7fa7ced1c640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7c8125a00 con 0x7fa7c80ac5a0 2024-03-29T10:49:10.810 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:10.823+0000 7fa7ced1c640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7c8125ca0 con 0x7fa7c80b0770 2024-03-29T10:49:10.810 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:10.823+0000 7fa7ced1c640 10 monclient(hunting): _renew_subs 2024-03-29T10:49:10.810 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:10.823+0000 7fa7ced1c640 10 monclient(hunting): authenticate will time out at 1273.058998s 2024-03-29T10:49:10.810 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:10.823+0000 7fa7c7fff640 1 --1- >> v1:172.21.15.88:6790/0 conn(0x7fa7c80b0770 0x7fa7c81440f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6790/0 says I am v1:172.21.15.88:51618/0 (socket says 172.21.15.88:51618) 2024-03-29T10:49:10.810 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:10.823+0000 7fa7c7fff640 1 -- 172.21.15.88:0/554643835 learned_addr learned my addr 172.21.15.88:0/554643835 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-29T10:49:10.810 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:10.823+0000 7fa7c57fa640 10 client.?.objecter ms_handle_connect 0x7fa7c80b0770 2024-03-29T10:49:10.810 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:10.823+0000 7fa7c57fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:49:10.810 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:10.823+0000 7fa7ccab8640 1 -- 172.21.15.88:0/554643835 >> v1:172.21.15.196:6789/0 conn(0x7fa7c80ac5a0 legacy=0x7fa7c81db710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-03-29T10:49:10.810 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:10.823+0000 7fa7ccab8640 1 -- 172.21.15.88:0/554643835 >> v1:172.21.15.196:6789/0 conn(0x7fa7c80ac5a0 legacy=0x7fa7c81db710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-29T10:49:10.810 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:10.823+0000 7fa7ccab8640 1 --1- 172.21.15.88:0/554643835 >> v1:172.21.15.196:6789/0 conn(0x7fa7c80ac5a0 0x7fa7c81db710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify read banner and identify addresses failed 2024-03-29T10:49:10.810 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:10.823+0000 7fa7ccab8640 1 --1- 172.21.15.88:0/554643835 >> v1:172.21.15.196:6789/0 conn(0x7fa7c80ac5a0 0x7fa7c81db710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).fault on lossy channel, failing 2024-03-29T10:49:10.810 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:10.823+0000 7fa7c57fa640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.196:6789/0 2024-03-29T10:49:10.810 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:10.823+0000 7fa7c57fa640 10 client.?.objecter ms_handle_connect 0x7fa7c814a700 2024-03-29T10:49:10.810 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:10.823+0000 7fa7c57fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:49:10.811 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:10.823+0000 7fa7c57fa640 1 -- 172.21.15.88:0/554643835 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3072462250 0 0) 0x7fa7c8125ca0 con 0x7fa7c80b0770 2024-03-29T10:49:10.811 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:10.824+0000 7fa7c57fa640 10 monclient(hunting): _negotiate 2024-03-29T10:49:10.811 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:10.824+0000 7fa7c57fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:10.811 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:10.824+0000 7fa7c57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:10.811 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:10.824+0000 7fa7c57fa640 10 monclient(hunting): my global_id is 4559 2024-03-29T10:49:10.811 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:10.824+0000 7fa7c57fa640 1 -- 172.21.15.88:0/554643835 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa7b4004eb0 con 0x7fa7c80b0770 2024-03-29T10:49:10.811 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:10.824+0000 7fa7c57fa640 1 -- 172.21.15.88:0/554643835 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3678713827 0 0) 0x7fa7b4004eb0 con 0x7fa7c80b0770 2024-03-29T10:49:10.811 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:10.824+0000 7fa7c57fa640 1 -- 172.21.15.88:0/554643835 --> v1:172.21.15.88:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa7c8125ca0 con 0x7fa7c80b0770 2024-03-29T10:49:10.811 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:10.825+0000 7fa7c57fa640 1 -- 172.21.15.88:0/554643835 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7fa7bc001b60 con 0x7fa7c80b0770 2024-03-29T10:49:10.811 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:10.825+0000 7fa7c57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:10.811 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:10.825+0000 7fa7c57fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:49:10.811 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:10.825+0000 7fa7c57fa640 10 monclient(hunting): dump: 2024-03-29T10:49:10.811 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:49:10.811 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:49:10.811 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:49:10.812 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:49:10.812 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:10.812 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:49:10.812 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:49:10.812 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:49:10.812 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:49:10.812 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:49:10.812 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:10.825+0000 7fa7c57fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:49:10.812 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:10.825+0000 7fa7c57fa640 1 -- 172.21.15.88:0/554643835 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4204214310 0 0) 0x7fa7c8127650 con 0x7fa7c814a700 2024-03-29T10:49:10.812 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:10.825+0000 7fa7c57fa640 10 monclient(hunting): _negotiate 2024-03-29T10:49:10.812 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:10.825+0000 7fa7c57fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:10.812 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:10.825+0000 7fa7c57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:10.813 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:10.825+0000 7fa7c57fa640 10 monclient(hunting): my global_id is 4536 2024-03-29T10:49:10.813 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:10.825+0000 7fa7c57fa640 1 -- 172.21.15.88:0/554643835 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa7b4004eb0 con 0x7fa7c814a700 2024-03-29T10:49:10.815 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:10.825+0000 7fa7c57fa640 1 -- 172.21.15.88:0/554643835 <== mon.2 v1:172.21.15.88:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3591936063 0 0) 0x7fa7c8125ca0 con 0x7fa7c80b0770 2024-03-29T10:49:10.815 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:10.825+0000 7fa7c57fa640 1 -- 172.21.15.88:0/554643835 >> v1:172.21.15.196:6789/0 conn(0x7fa7c80ac5a0 legacy=0x7fa7c81db710 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-29T10:49:10.815 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:10.825+0000 7fa7c57fa640 1 -- 172.21.15.88:0/554643835 >> v1:172.21.15.88:6789/0 conn(0x7fa7c814a700 legacy=0x7fa7c81e7030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:10.816 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:10.825+0000 7fa7c57fa640 10 monclient: _finish_hunting 0 2024-03-29T10:49:10.816 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:10.825+0000 7fa7c57fa640 1 monclient: found mon.c 2024-03-29T10:49:10.816 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:10.825+0000 7fa7c57fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:49:10.816 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:10.825+0000 7fa7c57fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-03-29T10:49:10.816 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:10.825+0000 7fa7c57fa640 1 -- 172.21.15.88:0/554643835 --> v1:172.21.15.88:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7c81dbe20 con 0x7fa7c80b0770 2024-03-29T10:49:10.816 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:10.825+0000 7fa7c57fa640 10 monclient: _finish_auth 0 2024-03-29T10:49:10.816 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:10.825+0000 7fa7c57fa640 10 monclient: _check_auth_tickets 2024-03-29T10:49:10.816 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:10.825+0000 7fa7c57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:49:10.816 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:10.825+0000 7fa7ced1c640 5 monclient: authenticate success, global_id 4559 2024-03-29T10:49:10.816 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:10.825+0000 7fa7ced1c640 10 monclient: _renew_subs 2024-03-29T10:49:10.816 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:10.825+0000 7fa7ced1c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-03-29T10:49:10.817 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:10.825+0000 7fa7ced1c640 1 -- 172.21.15.88:0/554643835 --> v1:172.21.15.88:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7c81d9250 con 0x7fa7c80b0770 2024-03-29T10:49:10.817 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:10.825+0000 7fa7ced1c640 10 client.4559.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:49:10.817 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:10.825+0000 7fa7ced1c640 10 monclient: _renew_subs 2024-03-29T10:49:10.817 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:10.825+0000 7fa7ced1c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-03-29T10:49:10.817 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:10.825+0000 7fa7ced1c640 1 -- 172.21.15.88:0/554643835 --> v1:172.21.15.88:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7c81449e0 con 0x7fa7c80b0770 2024-03-29T10:49:10.817 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:10.826+0000 7fa7ced1c640 1 librados: init done 2024-03-29T10:49:10.817 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:10.826+0000 7fa7c57fa640 1 -- 172.21.15.88:0/554643835 <== mon.2 v1:172.21.15.88:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa7bc004a90 con 0x7fa7c80b0770 2024-03-29T10:49:10.817 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:10.826+0000 7fa7c57fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:49:10.817 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:10.826+0000 7fa7c57fa640 1 -- 172.21.15.88:0/554643835 <== mon.2 v1:172.21.15.88:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7fa7bc004da0 con 0x7fa7c80b0770 2024-03-29T10:49:10.817 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:10.826+0000 7fa7c57fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:10.817 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:10.826+0000 7fa7c6ffd640 4 set_mon_vals no callback set 2024-03-29T10:49:10.817 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:10.826+0000 7fa7c57fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:49:10.818 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:10.826+0000 7fa7c57fa640 10 monclient: dump: 2024-03-29T10:49:10.818 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:49:10.818 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:49:10.818 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:49:10.818 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:49:10.818 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:10.818 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:49:10.818 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:49:10.818 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:49:10.818 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:49:10.818 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:49:10.819 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:10.827+0000 7fa7ced1c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:49:10.819 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:10.827+0000 7fa7ced1c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:49:10.819 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:10.828+0000 7fa7ced1c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-03-29T10:49:10.819 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:10.828+0000 7fa7ced1c640 1 -- 172.21.15.88:0/554643835 --> v1:172.21.15.88:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7a8005080 con 0x7fa7c80b0770 2024-03-29T10:49:10.819 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:10.830+0000 7fa7c57fa640 1 -- 172.21.15.88:0/554643835 <== mon.2 v1:172.21.15.88:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4235629220 0 0) 0x7fa7bc01ad70 con 0x7fa7c80b0770 2024-03-29T10:49:10.819 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:10.831+0000 7fa7ccab8640 10 monclient: get_auth_request con 0x7fa7b406ddb0 auth_method 0 2024-03-29T10:49:10.819 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:10.831+0000 7fa7c57fa640 1 -- 172.21.15.88:0/554643835 <== mon.2 v1:172.21.15.88:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4257673582 0 0) 0x7fa7bc084a20 con 0x7fa7c80b0770 2024-03-29T10:49:10.819 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:10.831+0000 7fa7c57fa640 10 client.4559.objecter ms_dispatch 0x7fa7c8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:49:10.819 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:10.831+0000 7fa7c57fa640 3 client.4559.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:49:10.819 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:10.831+0000 7fa7c57fa640 3 client.4559.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:49:10.819 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:10.831+0000 7fa7c57fa640 20 client.4559.objecter dump_active .. 0 homeless 2024-03-29T10:49:10.819 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:10.831+0000 7fa7c57fa640 10 client.4559.objecter ms_handle_connect 0x7fa7b406ddb0 2024-03-29T10:49:10.821 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:10.839+0000 7fa7c57fa640 1 -- 172.21.15.88:0/554643835 <== mon.2 v1:172.21.15.88:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa7bc050300 con 0x7fa7c80b0770 2024-03-29T10:49:10.821 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:10.839+0000 7fa7c57fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:49:10.821 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:10.839+0000 7fa7c57fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:49:11.174 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:11.192+0000 7fa7ced1c640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:49:11.174 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:11.192+0000 7fa7ced1c640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:49:11.175 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:11.192+0000 7fa7ced1c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-03-29T10:49:11.175 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:11.192+0000 7fa7ced1c640 1 -- 172.21.15.88:0/554643835 --> v1:172.21.15.88:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa7a8005330 con 0x7fa7c80b0770 2024-03-29T10:49:11.175 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:11.193+0000 7fa7c57fa640 1 -- 172.21.15.88:0/554643835 <== mon.2 v1:172.21.15.88:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 344506374) 0x7fa7bc052230 con 0x7fa7c80b0770 2024-03-29T10:49:11.175 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:11.193+0000 7fa7c57fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:49:11.175 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:11.193+0000 7fa7c57fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:49:11.176 INFO:teuthology.misc.health.smithi088.stdout: 2024-03-29T10:49:11.176 INFO:teuthology.misc.health.smithi088.stdout:{"epoch":10,"fsid":"869f5188-f92b-42dd-a46d-8c646e0388a9","created":"2024-03-29T10:40:40.285087+0000","modified":"2024-03-29T10:40:42.536230+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:40:42.536230+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a26c0c1c-8de0-438c-a3f6-3df10ea61574","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5de24915-8dec-49b6-8b16-f3170197c591","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ac054af3-d589-4565-9644-e6af75975c53","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3ec4f045-c714-4a22-bd1a-e07330b45ab3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"cddb9867-8e30-4951-9b13-3d1e78b9c45a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dc6d9911-785d-432f-8b53-56a1ccab302f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"77c88437-b587-496e-8b4d-1824ea8b5305","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ce963d96-5a8e-44c3-99de-c0dbebe234ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:49:11.182 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:11.197+0000 7fa7ced1c640 10 librados: watch_flush enter 2024-03-29T10:49:11.183 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:11.197+0000 7fa7ced1c640 10 librados: watch_flush exit 2024-03-29T10:49:11.183 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:11.197+0000 7fa7ced1c640 20 client.4559.objecter shutdown clearing up homeless session... 2024-03-29T10:49:11.183 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:11.197+0000 7fa7ced1c640 10 client.4559.objecter successfully canceled tick 2024-03-29T10:49:11.183 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:11.199+0000 7fa7ced1c640 1 -- 172.21.15.88:0/554643835 >> v1:172.21.15.88:6816/1752861349 conn(0x7fa7b406ddb0 legacy=0x7fa7b4070270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:11.183 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:11.199+0000 7fa7ced1c640 10 monclient: shutdown 2024-03-29T10:49:11.183 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:11.199+0000 7fa7ced1c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:49:11.183 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:11.199+0000 7fa7ced1c640 1 -- 172.21.15.88:0/554643835 >> v1:172.21.15.88:6790/0 conn(0x7fa7c80b0770 legacy=0x7fa7c81440f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:11.183 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:11.199+0000 7fa7ced1c640 1 -- 172.21.15.88:0/554643835 shutdown_connections 2024-03-29T10:49:11.183 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:11.199+0000 7fa7ced1c640 1 -- 172.21.15.88:0/554643835 >> 172.21.15.88:0/554643835 conn(0x7fa7c806b660 msgr2=0x7fa7c8149510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:49:11.183 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:11.199+0000 7fa7ced1c640 1 -- 172.21.15.88:0/554643835 shutdown_connections 2024-03-29T10:49:11.183 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:11.199+0000 7fa7ced1c640 1 -- 172.21.15.88:0/554643835 wait complete. 2024-03-29T10:49:11.183 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:11.199+0000 7fa7ced1c640 1 librados: shutdown 2024-03-29T10:49:11.199 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:49:18.006 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:49:18.155 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:18.173+0000 7f25eacb3640 10 monclient: get_monmap_and_config 2024-03-29T10:49:18.155 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:18.173+0000 7f25eacb3640 10 monclient: build_initial_monmap 2024-03-29T10:49:18.155 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:18.173+0000 7f25eacb3640 10 monclient: monmap: 2024-03-29T10:49:18.155 INFO:teuthology.misc.health.smithi088.stderr:epoch 0 2024-03-29T10:49:18.155 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:49:18.155 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:49:18.174172+0000 2024-03-29T10:49:18.156 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:49:18.174172+0000 2024-03-29T10:49:18.156 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 0 (unknown) 2024-03-29T10:49:18.156 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:49:18.156 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-03-29T10:49:18.156 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-29T10:49:18.156 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-b 2024-03-29T10:49:18.156 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:49:18.157 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:18.174+0000 7f25eacb3640 1 Processor -- start 2024-03-29T10:49:18.157 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:18.174+0000 7f25eacb3640 1 -- start start 2024-03-29T10:49:18.158 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:18.174+0000 7f25eacb3640 10 monclient: init 2024-03-29T10:49:18.158 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:18.175+0000 7f25eacb3640 10 monclient: _reopen_session rank -1 2024-03-29T10:49:18.158 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:18.175+0000 7f25eacb3640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:49:18.158 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:18.175+0000 7f25eacb3640 10 monclient(hunting): picked mon.noname-b con 0x7f25e41457f0 addr v1:172.21.15.88:6790/0 2024-03-29T10:49:18.158 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:18.175+0000 7f25eacb3640 10 monclient(hunting): picked mon.noname-c con 0x7f25e413f650 addr v1:172.21.15.196:6789/0 2024-03-29T10:49:18.158 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:18.175+0000 7f25eacb3640 10 monclient(hunting): picked mon.noname-a con 0x7f25e40a9d70 addr v1:172.21.15.88:6789/0 2024-03-29T10:49:18.158 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:18.175+0000 7f25eacb3640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25e40aaef0 con 0x7f25e40a9d70 2024-03-29T10:49:18.158 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:18.175+0000 7f25eacb3640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25e4145bd0 con 0x7f25e413f650 2024-03-29T10:49:18.158 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:18.175+0000 7f25e8a4f640 1 --1- >> v1:172.21.15.88:6790/0 conn(0x7f25e41457f0 0x7f25e413ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6790/0 says I am v1:172.21.15.88:39152/0 (socket says 172.21.15.88:39152) 2024-03-29T10:49:18.158 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:18.175+0000 7f25e8a4f640 1 -- 172.21.15.88:0/2334882959 learned_addr learned my addr 172.21.15.88:0/2334882959 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-29T10:49:18.158 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:18.175+0000 7f25eacb3640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25e4145eb0 con 0x7f25e41457f0 2024-03-29T10:49:18.158 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:18.175+0000 7f25eacb3640 10 monclient(hunting): _renew_subs 2024-03-29T10:49:18.159 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:18.175+0000 7f25eacb3640 10 monclient(hunting): authenticate will time out at 1280.411470s 2024-03-29T10:49:18.159 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:18.175+0000 7f25e37fe640 1 -- 172.21.15.88:0/2334882959 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3716107114 0 0) 0x7f25e4145eb0 con 0x7f25e41457f0 2024-03-29T10:49:18.159 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:18.175+0000 7f25e37fe640 10 monclient(hunting): _negotiate 2024-03-29T10:49:18.159 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:18.175+0000 7f25e37fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:18.159 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:18.175+0000 7f25e37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:18.159 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:18.175+0000 7f25e37fe640 10 monclient(hunting): my global_id is 4562 2024-03-29T10:49:18.159 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:18.175+0000 7f25e37fe640 1 -- 172.21.15.88:0/2334882959 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f25d4003520 con 0x7f25e41457f0 2024-03-29T10:49:18.162 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:18.176+0000 7f25e37fe640 1 -- 172.21.15.88:0/2334882959 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2973941217 0 0) 0x7f25e40aaef0 con 0x7f25e40a9d70 2024-03-29T10:49:18.162 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:18.176+0000 7f25e37fe640 10 monclient(hunting): _negotiate 2024-03-29T10:49:18.162 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:18.176+0000 7f25e37fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:18.162 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:18.176+0000 7f25e37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:18.162 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:18.176+0000 7f25e37fe640 10 monclient(hunting): my global_id is 4539 2024-03-29T10:49:18.162 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:18.176+0000 7f25e37fe640 1 -- 172.21.15.88:0/2334882959 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f25d4003840 con 0x7f25e40a9d70 2024-03-29T10:49:18.162 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:18.176+0000 7f25e37fe640 1 -- 172.21.15.88:0/2334882959 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1879636790 0 0) 0x7f25d4003520 con 0x7f25e41457f0 2024-03-29T10:49:18.162 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:18.176+0000 7f25e37fe640 1 -- 172.21.15.88:0/2334882959 >> v1:172.21.15.196:6789/0 conn(0x7f25e413f650 legacy=0x7f25e40b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:18.162 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:18.176+0000 7f25e37fe640 1 -- 172.21.15.88:0/2334882959 >> v1:172.21.15.88:6789/0 conn(0x7f25e40a9d70 legacy=0x7f25e414d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:18.162 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:18.176+0000 7f25e37fe640 10 monclient: _finish_hunting 0 2024-03-29T10:49:18.162 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:18.176+0000 7f25e37fe640 1 monclient: found mon.noname-b 2024-03-29T10:49:18.163 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:18.176+0000 7f25e37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:49:18.163 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:18.176+0000 7f25e37fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.88:6790/0 2024-03-29T10:49:18.163 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:18.176+0000 7f25e37fe640 1 -- 172.21.15.88:0/2334882959 --> v1:172.21.15.88:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25e4146340 con 0x7f25e41457f0 2024-03-29T10:49:18.163 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:18.176+0000 7f25e37fe640 10 monclient: _finish_auth 0 2024-03-29T10:49:18.163 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:18.176+0000 7f25e37fe640 10 monclient: _check_auth_tickets 2024-03-29T10:49:18.163 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:18.176+0000 7f25e37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:49:18.163 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:18.176+0000 7f25e37fe640 1 -- 172.21.15.88:0/2334882959 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f25d8001670 con 0x7f25e41457f0 2024-03-29T10:49:18.163 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:18.176+0000 7f25e37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:18.163 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:18.176+0000 7f25e37fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:49:18.163 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:18.177+0000 7f25e37fe640 10 monclient: dump: 2024-03-29T10:49:18.163 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:49:18.163 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:49:18.163 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:49:18.164 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:49:18.164 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:18.164 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:49:18.164 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:49:18.164 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:49:18.164 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:49:18.164 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:49:18.164 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:18.177+0000 7f25eacb3640 5 monclient: authenticate success, global_id 4562 2024-03-29T10:49:18.164 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:18.177+0000 7f25eacb3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:49:18.164 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:18.177+0000 7f25e37fe640 1 -- 172.21.15.88:0/2334882959 <== mon.2 v1:172.21.15.88:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f25d80018c0 con 0x7f25e41457f0 2024-03-29T10:49:18.164 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:18.177+0000 7f25e37fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:49:18.164 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:18.177+0000 7f25e37fe640 1 -- 172.21.15.88:0/2334882959 <== mon.2 v1:172.21.15.88:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f25d8001c50 con 0x7f25e41457f0 2024-03-29T10:49:18.164 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:18.177+0000 7f25e37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:18.164 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:18.177+0000 7f25e37fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:49:18.165 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:18.177+0000 7f25e37fe640 10 monclient: dump: 2024-03-29T10:49:18.165 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:49:18.165 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:49:18.165 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:49:18.165 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:49:18.165 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:18.165 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:49:18.165 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:49:18.165 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:49:18.165 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:49:18.165 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:49:18.165 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:18.177+0000 7f25eacb3640 10 monclient: get_monmap_and_config success 2024-03-29T10:49:18.165 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:18.177+0000 7f25eacb3640 4 set_mon_vals no callback set 2024-03-29T10:49:18.165 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:18.178+0000 7f25eacb3640 10 monclient: shutdown 2024-03-29T10:49:18.166 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:18.178+0000 7f25eacb3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:49:18.166 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:18.178+0000 7f25eacb3640 1 -- 172.21.15.88:0/2334882959 >> v1:172.21.15.88:6790/0 conn(0x7f25e41457f0 legacy=0x7f25e413ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:18.166 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:18.178+0000 7f25eacb3640 1 -- 172.21.15.88:0/2334882959 shutdown_connections 2024-03-29T10:49:18.166 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:18.178+0000 7f25eacb3640 1 -- 172.21.15.88:0/2334882959 >> 172.21.15.88:0/2334882959 conn(0x7f25e4066a00 msgr2=0x7f25e4066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:49:18.166 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:18.178+0000 7f25eacb3640 1 -- 172.21.15.88:0/2334882959 shutdown_connections 2024-03-29T10:49:18.166 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:18.178+0000 7f25eacb3640 1 -- 172.21.15.88:0/2334882959 wait complete. 2024-03-29T10:49:18.166 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:18.180+0000 7f25eacb3640 10 monclient: build_initial_monmap 2024-03-29T10:49:18.166 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:18.180+0000 7f25eacb3640 10 monclient: monmap: 2024-03-29T10:49:18.166 INFO:teuthology.misc.health.smithi088.stderr:epoch 0 2024-03-29T10:49:18.166 INFO:teuthology.misc.health.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:49:18.166 INFO:teuthology.misc.health.smithi088.stderr:last_changed 0.000000 2024-03-29T10:49:18.166 INFO:teuthology.misc.health.smithi088.stderr:created 0.000000 2024-03-29T10:49:18.166 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 0 (unknown) 2024-03-29T10:49:18.166 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:49:18.167 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-03-29T10:49:18.167 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-29T10:49:18.167 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-c 2024-03-29T10:49:18.167 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:49:18.172 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:18.181+0000 7f25eacb3640 1 librados: starting msgr at 2024-03-29T10:49:18.173 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:18.181+0000 7f25eacb3640 1 librados: starting objecter 2024-03-29T10:49:18.173 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:18.181+0000 7f25eacb3640 1 Processor -- start 2024-03-29T10:49:18.173 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:18.181+0000 7f25eacb3640 1 -- start start 2024-03-29T10:49:18.173 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:18.181+0000 7f25eacb3640 1 librados: setting wanted keys 2024-03-29T10:49:18.173 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:18.181+0000 7f25eacb3640 1 librados: calling monclient init 2024-03-29T10:49:18.173 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:18.181+0000 7f25eacb3640 10 monclient: init 2024-03-29T10:49:18.173 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:18.182+0000 7f25eacb3640 10 monclient: _reopen_session rank -1 2024-03-29T10:49:18.173 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:18.182+0000 7f25eacb3640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:49:18.173 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:18.182+0000 7f25eacb3640 10 monclient(hunting): picked mon.noname-a con 0x7f25e40a9d70 addr v1:172.21.15.88:6789/0 2024-03-29T10:49:18.173 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:18.182+0000 7f25e8a4f640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f25e40a9d70 0x7f25e413b6d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:47450/0 (socket says 172.21.15.88:47450) 2024-03-29T10:49:18.173 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:18.182+0000 7f25e8a4f640 1 -- 172.21.15.88:0/2124410276 learned_addr learned my addr 172.21.15.88:0/2124410276 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-29T10:49:18.173 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:18.182+0000 7f25eacb3640 10 monclient(hunting): picked mon.noname-c con 0x7f25e413f650 addr v1:172.21.15.88:6790/0 2024-03-29T10:49:18.174 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:18.182+0000 7f25e17fa640 10 client.?.objecter ms_handle_connect 0x7f25e40a9d70 2024-03-29T10:49:18.174 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:18.182+0000 7f25e17fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:49:18.174 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:18.182+0000 7f25eacb3640 10 monclient(hunting): picked mon.noname-b con 0x7f25e41457f0 addr v1:172.21.15.196:6789/0 2024-03-29T10:49:18.174 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:18.183+0000 7f25eacb3640 1 -- 172.21.15.88:0/2124410276 --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25e413bde0 con 0x7f25e40a9d70 2024-03-29T10:49:18.174 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:18.183+0000 7f25eacb3640 1 -- 172.21.15.88:0/2124410276 --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25e413aa10 con 0x7f25e41457f0 2024-03-29T10:49:18.174 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:18.183+0000 7f25eacb3640 1 -- 172.21.15.88:0/2124410276 --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25e413acf0 con 0x7f25e413f650 2024-03-29T10:49:18.174 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:18.183+0000 7f25eacb3640 10 monclient(hunting): _renew_subs 2024-03-29T10:49:18.174 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:18.183+0000 7f25eacb3640 10 monclient(hunting): authenticate will time out at 1280.419093s 2024-03-29T10:49:18.174 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:18.183+0000 7f25e17fa640 10 client.?.objecter ms_handle_connect 0x7f25e41457f0 2024-03-29T10:49:18.174 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:18.183+0000 7f25e17fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:49:18.174 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:18.183+0000 7f25e17fa640 1 -- 172.21.15.88:0/2124410276 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4280564947 0 0) 0x7f25e413bde0 con 0x7f25e40a9d70 2024-03-29T10:49:18.174 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:18.183+0000 7f25e17fa640 10 monclient(hunting): _negotiate 2024-03-29T10:49:18.174 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:18.183+0000 7f25e17fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:18.175 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:18.183+0000 7f25e17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:18.175 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:18.183+0000 7f25e17fa640 10 monclient(hunting): my global_id is 4542 2024-03-29T10:49:18.175 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:18.183+0000 7f25e17fa640 1 -- 172.21.15.88:0/2124410276 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f25c8004eb0 con 0x7f25e40a9d70 2024-03-29T10:49:18.175 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:18.183+0000 7f25e17fa640 10 client.?.objecter ms_handle_connect 0x7f25e413f650 2024-03-29T10:49:18.175 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:18.183+0000 7f25e17fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:49:18.175 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:18.183+0000 7f25e17fa640 1 -- 172.21.15.88:0/2124410276 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3517838569 0 0) 0x7f25e413aa10 con 0x7f25e41457f0 2024-03-29T10:49:18.175 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:18.183+0000 7f25e17fa640 10 monclient(hunting): _negotiate 2024-03-29T10:49:18.175 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:18.183+0000 7f25e17fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:18.175 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:18.183+0000 7f25e17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:18.175 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:18.183+0000 7f25e17fa640 10 monclient(hunting): my global_id is 4519 2024-03-29T10:49:18.175 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:18.183+0000 7f25e17fa640 1 -- 172.21.15.88:0/2124410276 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f25e413bde0 con 0x7f25e41457f0 2024-03-29T10:49:18.175 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:18.184+0000 7f25e17fa640 1 -- 172.21.15.88:0/2124410276 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 956261951 0 0) 0x7f25c8004eb0 con 0x7f25e40a9d70 2024-03-29T10:49:18.175 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:18.184+0000 7f25e17fa640 1 -- 172.21.15.88:0/2124410276 --> v1:172.21.15.88:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f25e413aa10 con 0x7f25e40a9d70 2024-03-29T10:49:18.175 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:18.184+0000 7f25e17fa640 1 -- 172.21.15.88:0/2124410276 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f25d8001c90 con 0x7f25e40a9d70 2024-03-29T10:49:18.175 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:18.184+0000 7f25e17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:18.176 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:18.184+0000 7f25e17fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:49:18.176 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:18.184+0000 7f25e17fa640 10 monclient(hunting): dump: 2024-03-29T10:49:18.176 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:49:18.176 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:49:18.176 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:49:18.176 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:49:18.176 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:18.176 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:49:18.176 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:49:18.176 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:49:18.176 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:49:18.176 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:49:18.176 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:18.184+0000 7f25e17fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:49:18.176 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:18.184+0000 7f25e17fa640 1 -- 172.21.15.88:0/2124410276 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3671009246 0 0) 0x7f25e413bde0 con 0x7f25e41457f0 2024-03-29T10:49:18.177 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:18.184+0000 7f25e17fa640 1 -- 172.21.15.88:0/2124410276 --> v1:172.21.15.196:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f25c8004eb0 con 0x7f25e41457f0 2024-03-29T10:49:18.177 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:18.184+0000 7f25e17fa640 1 -- 172.21.15.88:0/2124410276 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f25dc001cb0 con 0x7f25e41457f0 2024-03-29T10:49:18.177 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:18.184+0000 7f25e17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:18.177 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:18.184+0000 7f25e17fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T10:49:18.177 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:18.184+0000 7f25e17fa640 10 monclient(hunting): dump: 2024-03-29T10:49:18.177 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:49:18.177 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:49:18.177 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:49:18.177 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:49:18.177 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:18.177 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:49:18.177 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:49:18.177 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:49:18.177 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:49:18.178 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:49:18.178 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:18.184+0000 7f25e17fa640 1 -- 172.21.15.88:0/2124410276 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3438185939 0 0) 0x7f25e413acf0 con 0x7f25e413f650 2024-03-29T10:49:18.178 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:18.184+0000 7f25e17fa640 10 monclient(hunting): _negotiate 2024-03-29T10:49:18.178 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:18.184+0000 7f25e17fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:18.178 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:18.184+0000 7f25e17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:18.178 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:18.184+0000 7f25e17fa640 10 monclient(hunting): my global_id is 4565 2024-03-29T10:49:18.178 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:18.184+0000 7f25e17fa640 1 -- 172.21.15.88:0/2124410276 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f25e413bde0 con 0x7f25e413f650 2024-03-29T10:49:18.178 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:18.184+0000 7f25e17fa640 1 -- 172.21.15.88:0/2124410276 <== mon.1 v1:172.21.15.196:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 13022585 0 0) 0x7f25c8004eb0 con 0x7f25e41457f0 2024-03-29T10:49:18.178 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:18.184+0000 7f25e17fa640 1 -- 172.21.15.88:0/2124410276 >> v1:172.21.15.88:6790/0 conn(0x7f25e413f650 legacy=0x7f25e41405f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:18.178 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:18.184+0000 7f25e17fa640 1 -- 172.21.15.88:0/2124410276 >> v1:172.21.15.88:6789/0 conn(0x7f25e40a9d70 legacy=0x7f25e413b6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:18.178 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:18.184+0000 7f25e17fa640 10 monclient: _finish_hunting 0 2024-03-29T10:49:18.178 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:18.184+0000 7f25e17fa640 1 monclient: found mon.b 2024-03-29T10:49:18.178 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:18.184+0000 7f25e17fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:49:18.178 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:18.184+0000 7f25e17fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T10:49:18.179 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:18.184+0000 7f25e17fa640 1 -- 172.21.15.88:0/2124410276 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25e413afd0 con 0x7f25e41457f0 2024-03-29T10:49:18.179 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:18.184+0000 7f25e17fa640 10 monclient: _finish_auth 0 2024-03-29T10:49:18.179 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:18.184+0000 7f25e17fa640 10 monclient: _check_auth_tickets 2024-03-29T10:49:18.179 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:18.184+0000 7f25e17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:49:18.179 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:18.184+0000 7f25eacb3640 5 monclient: authenticate success, global_id 4519 2024-03-29T10:49:18.179 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:18.184+0000 7f25eacb3640 10 monclient: _renew_subs 2024-03-29T10:49:18.179 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:18.184+0000 7f25eacb3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T10:49:18.179 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:18.184+0000 7f25eacb3640 1 -- 172.21.15.88:0/2124410276 --> v1:172.21.15.196:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25e4135ae0 con 0x7f25e41457f0 2024-03-29T10:49:18.179 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:18.184+0000 7f25eacb3640 10 client.4519.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:49:18.179 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:18.184+0000 7f25eacb3640 10 monclient: _renew_subs 2024-03-29T10:49:18.179 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:18.184+0000 7f25eacb3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T10:49:18.179 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:18.184+0000 7f25eacb3640 1 -- 172.21.15.88:0/2124410276 --> v1:172.21.15.196:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f25e41e76e0 con 0x7f25e41457f0 2024-03-29T10:49:18.179 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:18.185+0000 7f25eacb3640 1 librados: init done 2024-03-29T10:49:18.180 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:18.185+0000 7f25e17fa640 1 -- 172.21.15.88:0/2124410276 <== mon.1 v1:172.21.15.196:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f25dc0020c0 con 0x7f25e41457f0 2024-03-29T10:49:18.180 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:18.185+0000 7f25e17fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:49:18.180 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:18.185+0000 7f25e17fa640 1 -- 172.21.15.88:0/2124410276 <== mon.1 v1:172.21.15.196:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f25dc003b10 con 0x7f25e41457f0 2024-03-29T10:49:18.180 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:18.185+0000 7f25e17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:18.180 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:18.185+0000 7f25e17fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:49:18.180 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:18.185+0000 7f25e17fa640 10 monclient: dump: 2024-03-29T10:49:18.180 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:49:18.180 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:49:18.180 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:49:18.180 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:49:18.180 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:18.180 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:49:18.180 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:49:18.180 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:49:18.181 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:49:18.181 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:49:18.181 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:18.185+0000 7f25e27fc640 4 set_mon_vals no callback set 2024-03-29T10:49:18.181 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:18.187+0000 7f25eacb3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:49:18.181 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:18.187+0000 7f25eacb3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:49:18.181 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:18.187+0000 7f25eacb3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T10:49:18.181 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:18.187+0000 7f25eacb3640 1 -- 172.21.15.88:0/2124410276 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25c4005080 con 0x7f25e41457f0 2024-03-29T10:49:18.181 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:18.188+0000 7f25e17fa640 1 -- 172.21.15.88:0/2124410276 <== mon.1 v1:172.21.15.196:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4235629220 0 0) 0x7f25dc019ae0 con 0x7f25e41457f0 2024-03-29T10:49:18.181 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:18.189+0000 7f25e17fa640 1 -- 172.21.15.88:0/2124410276 <== mon.1 v1:172.21.15.196:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4257673582 0 0) 0x7f25dc083af0 con 0x7f25e41457f0 2024-03-29T10:49:18.181 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:18.189+0000 7f25e17fa640 10 client.4519.objecter ms_dispatch 0x7f25e4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:49:18.181 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:18.189+0000 7f25e17fa640 3 client.4519.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:49:18.181 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:18.189+0000 7f25e17fa640 3 client.4519.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:49:18.181 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:18.189+0000 7f25e17fa640 20 client.4519.objecter dump_active .. 0 homeless 2024-03-29T10:49:18.181 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:18.189+0000 7f25e8a4f640 10 monclient: get_auth_request con 0x7f25c806dfa0 auth_method 0 2024-03-29T10:49:18.182 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:18.190+0000 7f25e17fa640 10 client.4519.objecter ms_handle_connect 0x7f25c806dfa0 2024-03-29T10:49:18.182 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:18.196+0000 7f25e17fa640 1 -- 172.21.15.88:0/2124410276 <== mon.1 v1:172.21.15.196:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f25dc04f3d0 con 0x7f25e41457f0 2024-03-29T10:49:18.182 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:18.196+0000 7f25e17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:49:18.182 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:18.196+0000 7f25e17fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:49:18.542 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:18.560+0000 7f25eacb3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:49:18.543 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:18.560+0000 7f25eacb3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:49:18.543 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:18.560+0000 7f25eacb3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T10:49:18.543 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:18.560+0000 7f25eacb3640 1 -- 172.21.15.88:0/2124410276 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f25c4005330 con 0x7f25e41457f0 2024-03-29T10:49:18.543 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:18.561+0000 7f25e17fa640 1 -- 172.21.15.88:0/2124410276 <== mon.1 v1:172.21.15.196:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 344506374) 0x7f25dc051300 con 0x7f25e41457f0 2024-03-29T10:49:18.543 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:18.561+0000 7f25e17fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:49:18.544 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:18.561+0000 7f25e17fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:49:18.544 INFO:teuthology.misc.health.smithi088.stdout: 2024-03-29T10:49:18.544 INFO:teuthology.misc.health.smithi088.stdout:{"epoch":10,"fsid":"869f5188-f92b-42dd-a46d-8c646e0388a9","created":"2024-03-29T10:40:40.285087+0000","modified":"2024-03-29T10:40:42.536230+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:40:42.536230+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a26c0c1c-8de0-438c-a3f6-3df10ea61574","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5de24915-8dec-49b6-8b16-f3170197c591","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ac054af3-d589-4565-9644-e6af75975c53","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3ec4f045-c714-4a22-bd1a-e07330b45ab3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"cddb9867-8e30-4951-9b13-3d1e78b9c45a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dc6d9911-785d-432f-8b53-56a1ccab302f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"77c88437-b587-496e-8b4d-1824ea8b5305","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ce963d96-5a8e-44c3-99de-c0dbebe234ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:49:18.547 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:18.564+0000 7f25eacb3640 10 librados: watch_flush enter 2024-03-29T10:49:18.547 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:18.564+0000 7f25eacb3640 10 librados: watch_flush exit 2024-03-29T10:49:18.547 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:18.564+0000 7f25eacb3640 20 client.4519.objecter shutdown clearing up homeless session... 2024-03-29T10:49:18.548 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:18.564+0000 7f25eacb3640 10 client.4519.objecter successfully canceled tick 2024-03-29T10:49:18.548 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:18.565+0000 7f25eacb3640 1 -- 172.21.15.88:0/2124410276 >> v1:172.21.15.88:6816/1752861349 conn(0x7f25c806dfa0 legacy=0x7f25c8070460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:18.548 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:18.565+0000 7f25eacb3640 10 monclient: shutdown 2024-03-29T10:49:18.548 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:18.565+0000 7f25eacb3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:49:18.548 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:18.565+0000 7f25eacb3640 1 -- 172.21.15.88:0/2124410276 >> v1:172.21.15.196:6789/0 conn(0x7f25e41457f0 legacy=0x7f25e4140d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:18.548 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:18.565+0000 7f25eacb3640 1 -- 172.21.15.88:0/2124410276 shutdown_connections 2024-03-29T10:49:18.548 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:18.565+0000 7f25eacb3640 1 -- 172.21.15.88:0/2124410276 >> 172.21.15.88:0/2124410276 conn(0x7f25e4066a00 msgr2=0x7f25e40b2520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:49:18.548 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:18.565+0000 7f25eacb3640 1 -- 172.21.15.88:0/2124410276 shutdown_connections 2024-03-29T10:49:18.549 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:18.565+0000 7f25eacb3640 1 -- 172.21.15.88:0/2124410276 wait complete. 2024-03-29T10:49:18.549 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:18.565+0000 7f25eacb3640 1 librados: shutdown 2024-03-29T10:49:18.563 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:49:25.370 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:49:25.516 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:25.533+0000 7fafadfe3640 10 monclient: get_monmap_and_config 2024-03-29T10:49:25.516 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:25.533+0000 7fafadfe3640 10 monclient: build_initial_monmap 2024-03-29T10:49:25.516 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:25.533+0000 7fafadfe3640 10 monclient: monmap: 2024-03-29T10:49:25.516 INFO:teuthology.misc.health.smithi088.stderr:epoch 0 2024-03-29T10:49:25.516 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:49:25.516 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:49:25.534836+0000 2024-03-29T10:49:25.516 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:49:25.534836+0000 2024-03-29T10:49:25.516 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 0 (unknown) 2024-03-29T10:49:25.516 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:49:25.516 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-03-29T10:49:25.516 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-29T10:49:25.517 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-b 2024-03-29T10:49:25.517 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:49:25.517 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:25.534+0000 7fafadfe3640 1 Processor -- start 2024-03-29T10:49:25.517 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:25.535+0000 7fafadfe3640 1 -- start start 2024-03-29T10:49:25.517 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:25.535+0000 7fafadfe3640 10 monclient: init 2024-03-29T10:49:25.518 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:25.536+0000 7fafadfe3640 10 monclient: _reopen_session rank -1 2024-03-29T10:49:25.518 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:25.536+0000 7fafadfe3640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:49:25.521 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:25.536+0000 7fafadfe3640 10 monclient(hunting): picked mon.noname-b con 0x7fafa00389e0 addr v1:172.21.15.88:6790/0 2024-03-29T10:49:25.526 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:25.542+0000 7fafacfe1640 1 --1- >> v1:172.21.15.88:6790/0 conn(0x7fafa00389e0 0x7fafa0033520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6790/0 says I am v1:172.21.15.88:42493/0 (socket says 172.21.15.88:42493) 2024-03-29T10:49:25.526 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:25.542+0000 7fafa7fff640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fafa0038dc0 0x7fafa00cadd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:42389/0 (socket says 172.21.15.88:42389) 2024-03-29T10:49:25.526 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:25.543+0000 7fafadfe3640 10 monclient(hunting): picked mon.noname-a con 0x7fafa0038dc0 addr v1:172.21.15.88:6789/0 2024-03-29T10:49:25.526 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:25.543+0000 7fafacfe1640 1 -- 172.21.15.88:0/1473793686 learned_addr learned my addr 172.21.15.88:0/1473793686 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-29T10:49:25.526 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:25.543+0000 7fafadfe3640 10 monclient(hunting): picked mon.noname-c con 0x7fafa0030660 addr v1:172.21.15.196:6789/0 2024-03-29T10:49:25.526 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:25.543+0000 7fafadfe3640 1 -- 172.21.15.88:0/1473793686 --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafa00c54c0 con 0x7fafa0038dc0 2024-03-29T10:49:25.526 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:25.543+0000 7fafadfe3640 1 -- 172.21.15.88:0/1473793686 --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafa0036430 con 0x7fafa0030660 2024-03-29T10:49:25.526 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:25.543+0000 7fafadfe3640 1 -- 172.21.15.88:0/1473793686 --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafa0039840 con 0x7fafa00389e0 2024-03-29T10:49:25.526 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:25.543+0000 7fafadfe3640 10 monclient(hunting): _renew_subs 2024-03-29T10:49:25.526 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:25.543+0000 7fafadfe3640 10 monclient(hunting): authenticate will time out at 1287.779595s 2024-03-29T10:49:25.527 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:25.543+0000 7fafa77fe640 1 -- 172.21.15.88:0/1473793686 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3628069971 0 0) 0x7fafa0039840 con 0x7fafa00389e0 2024-03-29T10:49:25.527 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:25.543+0000 7fafa77fe640 10 monclient(hunting): _negotiate 2024-03-29T10:49:25.527 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:25.543+0000 7fafa77fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:25.527 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:25.543+0000 7fafa77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:25.527 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:25.543+0000 7fafa77fe640 10 monclient(hunting): my global_id is 4568 2024-03-29T10:49:25.527 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:25.544+0000 7fafa77fe640 1 -- 172.21.15.88:0/1473793686 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faf98003520 con 0x7fafa00389e0 2024-03-29T10:49:25.528 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:25.546+0000 7fafa77fe640 1 -- 172.21.15.88:0/1473793686 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2904819993 0 0) 0x7faf98003520 con 0x7fafa00389e0 2024-03-29T10:49:25.528 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:25.546+0000 7fafa77fe640 1 -- 172.21.15.88:0/1473793686 >> v1:172.21.15.196:6789/0 conn(0x7fafa0030660 legacy=0x7fafa00d3170 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:49:25.530 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:25.547+0000 7fafa77fe640 1 -- 172.21.15.88:0/1473793686 >> v1:172.21.15.88:6789/0 conn(0x7fafa0038dc0 legacy=0x7fafa00cadd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:25.530 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:25.547+0000 7fafa77fe640 10 monclient: _finish_hunting 0 2024-03-29T10:49:25.530 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:25.547+0000 7fafa77fe640 1 monclient: found mon.noname-b 2024-03-29T10:49:25.530 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:25.547+0000 7fafa77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:49:25.530 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:25.547+0000 7fafa77fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.88:6790/0 2024-03-29T10:49:25.530 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:25.547+0000 7fafa77fe640 1 -- 172.21.15.88:0/1473793686 --> v1:172.21.15.88:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafa0035690 con 0x7fafa00389e0 2024-03-29T10:49:25.530 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:25.547+0000 7fafa77fe640 10 monclient: _finish_auth 0 2024-03-29T10:49:25.530 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:25.547+0000 7fafa77fe640 10 monclient: _check_auth_tickets 2024-03-29T10:49:25.530 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:25.547+0000 7fafa77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:49:25.530 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:25.547+0000 7fafa77fe640 1 -- 172.21.15.88:0/1473793686 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7faf9c001690 con 0x7fafa00389e0 2024-03-29T10:49:25.531 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:25.547+0000 7fafadfe3640 5 monclient: authenticate success, global_id 4568 2024-03-29T10:49:25.531 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:25.547+0000 7fafadfe3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:49:25.531 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:25.547+0000 7fafa77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:25.531 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:25.547+0000 7fafa77fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:49:25.531 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:25.547+0000 7fafa77fe640 10 monclient: dump: 2024-03-29T10:49:25.531 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:49:25.531 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:49:25.531 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:49:25.531 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:49:25.531 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:25.531 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:49:25.531 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:49:25.531 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:49:25.531 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:49:25.531 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:49:25.532 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:25.547+0000 7fafadfe3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:49:25.533 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:25.547+0000 7fafa77fe640 1 -- 172.21.15.88:0/1473793686 <== mon.2 v1:172.21.15.88:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7faf9c001860 con 0x7fafa00389e0 2024-03-29T10:49:25.533 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:25.547+0000 7fafa77fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:49:25.533 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:25.547+0000 7fafa77fe640 1 -- 172.21.15.88:0/1473793686 <== mon.2 v1:172.21.15.88:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7faf9c001cb0 con 0x7fafa00389e0 2024-03-29T10:49:25.533 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:25.547+0000 7fafadfe3640 10 monclient: get_monmap_and_config success 2024-03-29T10:49:25.533 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:25.547+0000 7fafadfe3640 4 set_mon_vals no callback set 2024-03-29T10:49:25.533 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:25.548+0000 7fafadfe3640 10 monclient: shutdown 2024-03-29T10:49:25.533 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:25.548+0000 7fafa77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:25.533 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:25.548+0000 7fafa77fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:49:25.533 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:25.548+0000 7fafa77fe640 10 monclient: dump: 2024-03-29T10:49:25.534 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:49:25.534 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:49:25.534 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:49:25.534 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:49:25.534 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:25.534 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:49:25.534 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:49:25.534 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:49:25.534 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:49:25.534 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:49:25.534 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:25.548+0000 7fafadfe3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:49:25.535 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:25.548+0000 7fafadfe3640 1 -- 172.21.15.88:0/1473793686 >> v1:172.21.15.88:6790/0 conn(0x7fafa00389e0 legacy=0x7fafa0033520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:25.535 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:25.549+0000 7fafadfe3640 1 -- 172.21.15.88:0/1473793686 shutdown_connections 2024-03-29T10:49:25.535 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:25.549+0000 7fafadfe3640 1 -- 172.21.15.88:0/1473793686 >> 172.21.15.88:0/1473793686 conn(0x7fafa0005c00 msgr2=0x7fafa0006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:49:25.535 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:25.552+0000 7fafadfe3640 1 -- 172.21.15.88:0/1473793686 shutdown_connections 2024-03-29T10:49:25.535 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:25.553+0000 7fafadfe3640 1 -- 172.21.15.88:0/1473793686 wait complete. 2024-03-29T10:49:25.536 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:25.554+0000 7fafadfe3640 10 monclient: build_initial_monmap 2024-03-29T10:49:25.536 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:25.554+0000 7fafadfe3640 10 monclient: monmap: 2024-03-29T10:49:25.536 INFO:teuthology.misc.health.smithi088.stderr:epoch 0 2024-03-29T10:49:25.536 INFO:teuthology.misc.health.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:49:25.536 INFO:teuthology.misc.health.smithi088.stderr:last_changed 0.000000 2024-03-29T10:49:25.536 INFO:teuthology.misc.health.smithi088.stderr:created 0.000000 2024-03-29T10:49:25.537 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 0 (unknown) 2024-03-29T10:49:25.537 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:49:25.537 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-03-29T10:49:25.537 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-29T10:49:25.537 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-c 2024-03-29T10:49:25.537 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:49:25.538 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:25.555+0000 7fafadfe3640 1 librados: starting msgr at 2024-03-29T10:49:25.538 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:25.555+0000 7fafadfe3640 1 librados: starting objecter 2024-03-29T10:49:25.538 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:25.555+0000 7fafadfe3640 1 Processor -- start 2024-03-29T10:49:25.538 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:25.556+0000 7fafadfe3640 1 -- start start 2024-03-29T10:49:25.542 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:25.556+0000 7fafadfe3640 1 librados: setting wanted keys 2024-03-29T10:49:25.542 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:25.556+0000 7fafadfe3640 1 librados: calling monclient init 2024-03-29T10:49:25.543 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:25.556+0000 7fafadfe3640 10 monclient: init 2024-03-29T10:49:25.543 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:25.557+0000 7fafadfe3640 10 monclient: _reopen_session rank -1 2024-03-29T10:49:25.544 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:25.559+0000 7fafadfe3640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:49:25.544 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:25.559+0000 7fafadfe3640 10 monclient(hunting): picked mon.noname-c con 0x7fafa0030660 addr v1:172.21.15.88:6790/0 2024-03-29T10:49:25.544 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:25.559+0000 7fafadfe3640 10 monclient(hunting): picked mon.noname-a con 0x7fafa00389e0 addr v1:172.21.15.88:6789/0 2024-03-29T10:49:25.544 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:25.560+0000 7fafadfe3640 10 monclient(hunting): picked mon.noname-b con 0x7fafa002d150 addr v1:172.21.15.196:6789/0 2024-03-29T10:49:25.544 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:25.560+0000 7fafadfe3640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafa00c7ea0 con 0x7fafa00389e0 2024-03-29T10:49:25.544 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:25.560+0000 7fafadfe3640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafa00c83b0 con 0x7fafa002d150 2024-03-29T10:49:25.544 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:25.560+0000 7fafadfe3640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafa00c5650 con 0x7fafa0030660 2024-03-29T10:49:25.544 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:25.560+0000 7fafadfe3640 10 monclient(hunting): _renew_subs 2024-03-29T10:49:25.544 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:25.560+0000 7fafadfe3640 10 monclient(hunting): authenticate will time out at 1287.796072s 2024-03-29T10:49:25.547 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:25.566+0000 7fafad7e2640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7fafa002d150 0x7fafa016b690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.88:33497/0 (socket says 172.21.15.88:33497) 2024-03-29T10:49:25.549 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:25.566+0000 7fafa7fff640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fafa00389e0 0x7fafa0164ad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:42573/0 (socket says 172.21.15.88:42573) 2024-03-29T10:49:25.549 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:25.566+0000 7fafa7fff640 1 -- 172.21.15.88:0/3109180492 learned_addr learned my addr 172.21.15.88:0/3109180492 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-29T10:49:25.549 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:25.566+0000 7fafa57fa640 10 client.?.objecter ms_handle_connect 0x7fafa00389e0 2024-03-29T10:49:25.549 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:25.566+0000 7fafa57fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:49:25.550 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:25.566+0000 7fafa57fa640 1 -- 172.21.15.88:0/3109180492 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3374310722 0 0) 0x7fafa00c7ea0 con 0x7fafa00389e0 2024-03-29T10:49:25.550 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:25.566+0000 7fafa57fa640 10 monclient(hunting): _negotiate 2024-03-29T10:49:25.550 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:25.566+0000 7fafa57fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:25.550 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:25.566+0000 7fafa57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:25.550 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:25.566+0000 7fafa57fa640 10 monclient(hunting): my global_id is 4548 2024-03-29T10:49:25.550 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:25.566+0000 7fafa57fa640 1 -- 172.21.15.88:0/3109180492 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faf8c004eb0 con 0x7fafa00389e0 2024-03-29T10:49:25.550 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:25.567+0000 7fafa57fa640 1 -- 172.21.15.88:0/3109180492 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1883236962 0 0) 0x7faf8c004eb0 con 0x7fafa00389e0 2024-03-29T10:49:25.550 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:25.567+0000 7fafa57fa640 1 -- 172.21.15.88:0/3109180492 --> v1:172.21.15.88:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fafa00c7ea0 con 0x7fafa00389e0 2024-03-29T10:49:25.550 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:25.567+0000 7fafa57fa640 1 -- 172.21.15.88:0/3109180492 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7faf94001930 con 0x7fafa00389e0 2024-03-29T10:49:25.550 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:25.567+0000 7fafa57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:25.550 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:25.567+0000 7fafa57fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:49:25.550 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:25.567+0000 7fafa57fa640 10 monclient(hunting): dump: 2024-03-29T10:49:25.551 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:49:25.551 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:49:25.551 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:49:25.551 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:49:25.551 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:25.551 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:49:25.551 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:49:25.551 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:49:25.551 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:49:25.551 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:49:25.551 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:25.567+0000 7fafa57fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:49:25.551 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:25.567+0000 7fafa57fa640 10 client.?.objecter ms_handle_connect 0x7fafa002d150 2024-03-29T10:49:25.551 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:25.567+0000 7fafa57fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:49:25.552 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:25.568+0000 7fafa57fa640 1 -- 172.21.15.88:0/3109180492 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1562189171 0 0) 0x7fafa00c83b0 con 0x7fafa002d150 2024-03-29T10:49:25.552 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:25.568+0000 7fafa57fa640 10 monclient(hunting): _negotiate 2024-03-29T10:49:25.552 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:25.568+0000 7fafa57fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:25.552 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:25.568+0000 7fafa57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:25.552 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:25.568+0000 7fafa57fa640 10 monclient(hunting): my global_id is 4522 2024-03-29T10:49:25.552 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:25.568+0000 7fafa57fa640 1 -- 172.21.15.88:0/3109180492 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faf8c0056d0 con 0x7fafa002d150 2024-03-29T10:49:25.552 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:25.569+0000 7fafa57fa640 1 -- 172.21.15.88:0/3109180492 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4190263928 0 0) 0x7faf8c0056d0 con 0x7fafa002d150 2024-03-29T10:49:25.553 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:25.569+0000 7fafa57fa640 1 -- 172.21.15.88:0/3109180492 --> v1:172.21.15.196:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7faf8c005b60 con 0x7fafa002d150 2024-03-29T10:49:25.553 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:25.569+0000 7fafa57fa640 10 client.?.objecter ms_handle_connect 0x7fafa0030660 2024-03-29T10:49:25.553 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:25.569+0000 7fafa57fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:49:25.553 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:25.569+0000 7fafa57fa640 1 -- 172.21.15.88:0/3109180492 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7fafa804fdd0 con 0x7fafa002d150 2024-03-29T10:49:25.553 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:25.569+0000 7fafa57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:25.553 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:25.569+0000 7fafa57fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T10:49:25.553 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:25.569+0000 7fafa57fa640 10 monclient(hunting): dump: 2024-03-29T10:49:25.553 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:49:25.553 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:49:25.553 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:49:25.553 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:49:25.553 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:25.553 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:49:25.553 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:49:25.554 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:49:25.554 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:49:25.554 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:49:25.554 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:25.569+0000 7fafa57fa640 1 -- 172.21.15.88:0/3109180492 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3308116847 0 0) 0x7fafa00c5650 con 0x7fafa0030660 2024-03-29T10:49:25.554 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:25.569+0000 7fafa57fa640 10 monclient(hunting): _negotiate 2024-03-29T10:49:25.554 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:25.569+0000 7fafa57fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:25.554 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:25.569+0000 7fafa57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:25.554 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:25.569+0000 7fafa57fa640 10 monclient(hunting): my global_id is 4571 2024-03-29T10:49:25.554 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:25.570+0000 7fafa57fa640 1 -- 172.21.15.88:0/3109180492 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faf8c005f90 con 0x7fafa0030660 2024-03-29T10:49:25.557 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:25.570+0000 7fafa57fa640 1 -- 172.21.15.88:0/3109180492 <== mon.0 v1:172.21.15.88:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2310202575 0 0) 0x7fafa00c7ea0 con 0x7fafa00389e0 2024-03-29T10:49:25.557 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:25.570+0000 7fafa57fa640 1 -- 172.21.15.88:0/3109180492 >> v1:172.21.15.88:6790/0 conn(0x7fafa0030660 legacy=0x7fafa01643c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:25.557 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:25.570+0000 7fafa57fa640 1 -- 172.21.15.88:0/3109180492 >> v1:172.21.15.196:6789/0 conn(0x7fafa002d150 legacy=0x7fafa016b690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:25.557 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:25.570+0000 7fafa57fa640 10 monclient: _finish_hunting 0 2024-03-29T10:49:25.557 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:25.570+0000 7fafa57fa640 1 monclient: found mon.a 2024-03-29T10:49:25.558 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:25.570+0000 7fafa57fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:49:25.558 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:25.570+0000 7fafa57fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-03-29T10:49:25.558 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:25.570+0000 7fafa57fa640 1 -- 172.21.15.88:0/3109180492 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafa00c57e0 con 0x7fafa00389e0 2024-03-29T10:49:25.558 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:25.570+0000 7fafa57fa640 10 monclient: _finish_auth 0 2024-03-29T10:49:25.558 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:25.570+0000 7fafa57fa640 10 monclient: _check_auth_tickets 2024-03-29T10:49:25.558 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:25.570+0000 7fafa57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:49:25.558 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:25.570+0000 7fafa57fa640 1 -- 172.21.15.88:0/3109180492 <== mon.0 v1:172.21.15.88:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7faf940047e0 con 0x7fafa00389e0 2024-03-29T10:49:25.558 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:25.570+0000 7fafa57fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:49:25.558 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:25.570+0000 7fafa67fc640 4 set_mon_vals no callback set 2024-03-29T10:49:25.558 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:25.571+0000 7fafadfe3640 5 monclient: authenticate success, global_id 4548 2024-03-29T10:49:25.558 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:25.571+0000 7fafadfe3640 10 monclient: _renew_subs 2024-03-29T10:49:25.558 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:25.571+0000 7fafadfe3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-03-29T10:49:25.558 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:25.571+0000 7fafadfe3640 1 -- 172.21.15.88:0/3109180492 --> v1:172.21.15.88:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fafa002fec0 con 0x7fafa00389e0 2024-03-29T10:49:25.558 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:25.571+0000 7fafa57fa640 1 -- 172.21.15.88:0/3109180492 <== mon.0 v1:172.21.15.88:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7faf94001930 con 0x7fafa00389e0 2024-03-29T10:49:25.559 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:25.571+0000 7fafa57fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:25.559 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:25.571+0000 7fafa57fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:49:25.559 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:25.571+0000 7fafa57fa640 10 monclient: dump: 2024-03-29T10:49:25.559 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:49:25.559 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:49:25.559 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:49:25.559 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:49:25.559 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:25.559 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:49:25.559 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:49:25.559 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:49:25.559 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:49:25.559 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:49:25.559 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:25.571+0000 7fafadfe3640 10 client.4548.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:49:25.560 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:25.571+0000 7fafadfe3640 10 monclient: _renew_subs 2024-03-29T10:49:25.560 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:25.571+0000 7fafadfe3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-03-29T10:49:25.560 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:25.571+0000 7fafadfe3640 1 -- 172.21.15.88:0/3109180492 --> v1:172.21.15.88:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fafa016cf90 con 0x7fafa00389e0 2024-03-29T10:49:25.560 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:25.571+0000 7fafadfe3640 1 librados: init done 2024-03-29T10:49:25.560 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:25.572+0000 7faf92ffd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:49:25.560 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:25.572+0000 7faf92ffd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:49:25.560 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:25.572+0000 7faf92ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-03-29T10:49:25.560 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:25.572+0000 7faf92ffd640 1 -- 172.21.15.88:0/3109180492 --> v1:172.21.15.88:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fafa0007b90 con 0x7fafa00389e0 2024-03-29T10:49:25.561 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:25.576+0000 7fafa57fa640 1 -- 172.21.15.88:0/3109180492 <== mon.0 v1:172.21.15.88:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4235629220 0 0) 0x7faf9401a7c0 con 0x7fafa00389e0 2024-03-29T10:49:25.561 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:25.577+0000 7fafa57fa640 1 -- 172.21.15.88:0/3109180492 <== mon.0 v1:172.21.15.88:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4257673582 0 0) 0x7faf94084790 con 0x7fafa00389e0 2024-03-29T10:49:25.561 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:25.577+0000 7fafa57fa640 10 client.4548.objecter ms_dispatch 0x7fafa8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:49:25.561 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:25.577+0000 7fafa57fa640 3 client.4548.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:49:25.561 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:25.577+0000 7fafa57fa640 3 client.4548.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:49:25.561 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:25.577+0000 7fafa57fa640 20 client.4548.objecter dump_active .. 0 homeless 2024-03-29T10:49:25.561 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:25.577+0000 7fafacfe1640 10 monclient: get_auth_request con 0x7faf8c06e680 auth_method 0 2024-03-29T10:49:25.562 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:25.578+0000 7fafa57fa640 10 client.4548.objecter ms_handle_connect 0x7faf8c06e680 2024-03-29T10:49:25.568 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:25.586+0000 7fafa57fa640 1 -- 172.21.15.88:0/3109180492 <== mon.0 v1:172.21.15.88:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7faf9404fea0 con 0x7fafa00389e0 2024-03-29T10:49:25.568 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:25.586+0000 7fafa57fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:49:25.568 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:25.586+0000 7fafa57fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:49:25.893 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:25.910+0000 7faf92ffd640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:49:25.893 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:25.910+0000 7faf92ffd640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:49:25.893 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:25.911+0000 7faf92ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-03-29T10:49:25.893 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:25.911+0000 7faf92ffd640 1 -- 172.21.15.88:0/3109180492 --> v1:172.21.15.88:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fafa0030050 con 0x7fafa00389e0 2024-03-29T10:49:25.896 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:25.914+0000 7fafa57fa640 1 -- 172.21.15.88:0/3109180492 <== mon.0 v1:172.21.15.88:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 344506374) 0x7faf9404fea0 con 0x7fafa00389e0 2024-03-29T10:49:25.896 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:25.914+0000 7fafa57fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:49:25.896 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:25.914+0000 7fafa57fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:49:25.896 INFO:teuthology.misc.health.smithi088.stdout: 2024-03-29T10:49:25.896 INFO:teuthology.misc.health.smithi088.stdout:{"epoch":10,"fsid":"869f5188-f92b-42dd-a46d-8c646e0388a9","created":"2024-03-29T10:40:40.285087+0000","modified":"2024-03-29T10:40:42.536230+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:40:42.536230+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a26c0c1c-8de0-438c-a3f6-3df10ea61574","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5de24915-8dec-49b6-8b16-f3170197c591","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ac054af3-d589-4565-9644-e6af75975c53","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3ec4f045-c714-4a22-bd1a-e07330b45ab3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"cddb9867-8e30-4951-9b13-3d1e78b9c45a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dc6d9911-785d-432f-8b53-56a1ccab302f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"77c88437-b587-496e-8b4d-1824ea8b5305","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ce963d96-5a8e-44c3-99de-c0dbebe234ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:49:25.900 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:25.917+0000 7faf92ffd640 10 librados: watch_flush enter 2024-03-29T10:49:25.900 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:25.918+0000 7faf92ffd640 10 librados: watch_flush exit 2024-03-29T10:49:25.900 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:25.918+0000 7faf92ffd640 20 client.4548.objecter shutdown clearing up homeless session... 2024-03-29T10:49:25.900 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:25.918+0000 7faf92ffd640 10 client.4548.objecter successfully canceled tick 2024-03-29T10:49:25.900 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:25.918+0000 7faf92ffd640 1 -- 172.21.15.88:0/3109180492 >> v1:172.21.15.88:6816/1752861349 conn(0x7faf8c06e680 legacy=0x7faf8c070b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:25.900 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:25.918+0000 7faf92ffd640 10 monclient: shutdown 2024-03-29T10:49:25.901 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:25.918+0000 7faf92ffd640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:49:25.901 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:25.918+0000 7faf92ffd640 1 -- 172.21.15.88:0/3109180492 >> v1:172.21.15.88:6789/0 conn(0x7fafa00389e0 legacy=0x7fafa0164ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:25.902 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:25.920+0000 7faf92ffd640 1 -- 172.21.15.88:0/3109180492 shutdown_connections 2024-03-29T10:49:25.902 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:25.920+0000 7faf92ffd640 1 -- 172.21.15.88:0/3109180492 >> 172.21.15.88:0/3109180492 conn(0x7fafa0005c00 msgr2=0x7fafa0034090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:49:25.902 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:25.920+0000 7faf92ffd640 1 -- 172.21.15.88:0/3109180492 shutdown_connections 2024-03-29T10:49:25.903 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:25.920+0000 7faf92ffd640 1 -- 172.21.15.88:0/3109180492 wait complete. 2024-03-29T10:49:25.903 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:25.920+0000 7faf92ffd640 1 librados: shutdown 2024-03-29T10:49:25.922 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:49:32.729 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:49:32.870 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:32.888+0000 7f35108e2640 10 monclient: get_monmap_and_config 2024-03-29T10:49:32.870 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:32.888+0000 7f35108e2640 10 monclient: build_initial_monmap 2024-03-29T10:49:32.870 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:32.888+0000 7f35108e2640 10 monclient: monmap: 2024-03-29T10:49:32.870 INFO:teuthology.misc.health.smithi088.stderr:epoch 0 2024-03-29T10:49:32.870 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:49:32.870 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:49:32.889140+0000 2024-03-29T10:49:32.870 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:49:32.889140+0000 2024-03-29T10:49:32.870 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 0 (unknown) 2024-03-29T10:49:32.870 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:49:32.871 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-03-29T10:49:32.871 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-29T10:49:32.871 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-b 2024-03-29T10:49:32.871 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:49:32.871 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:32.889+0000 7f35108e2640 1 Processor -- start 2024-03-29T10:49:32.871 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:32.889+0000 7f35108e2640 1 -- start start 2024-03-29T10:49:32.871 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:32.889+0000 7f35108e2640 10 monclient: init 2024-03-29T10:49:32.872 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:32.890+0000 7f35108e2640 10 monclient: _reopen_session rank -1 2024-03-29T10:49:32.872 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:32.890+0000 7f35108e2640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:49:32.872 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:32.890+0000 7f35108e2640 10 monclient(hunting): picked mon.noname-a con 0x7f350c0b0500 addr v1:172.21.15.88:6789/0 2024-03-29T10:49:32.872 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:32.890+0000 7f35108e2640 10 monclient(hunting): picked mon.noname-c con 0x7f350c0b08e0 addr v1:172.21.15.196:6789/0 2024-03-29T10:49:32.872 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:32.890+0000 7f35108e2640 10 monclient(hunting): picked mon.noname-b con 0x7f350c0aeca0 addr v1:172.21.15.88:6790/0 2024-03-29T10:49:32.872 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:32.890+0000 7f35108e2640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f350c140620 con 0x7f350c0b0500 2024-03-29T10:49:32.872 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:32.890+0000 7f35108e2640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f350c13daf0 con 0x7f350c0b08e0 2024-03-29T10:49:32.872 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:32.890+0000 7f35108e2640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f350c0ae180 con 0x7f350c0aeca0 2024-03-29T10:49:32.873 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:32.890+0000 7f35108e2640 10 monclient(hunting): _renew_subs 2024-03-29T10:49:32.873 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:32.890+0000 7f35108e2640 10 monclient(hunting): authenticate will time out at 1295.126466s 2024-03-29T10:49:32.873 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:32.890+0000 7f350ad9d640 1 --1- >> v1:172.21.15.88:6790/0 conn(0x7f350c0aeca0 0x7f350c14b7d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6790/0 says I am v1:172.21.15.88:52556/0 (socket says 172.21.15.88:52556) 2024-03-29T10:49:32.873 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:32.890+0000 7f350ad9d640 1 -- 172.21.15.88:0/2094333762 learned_addr learned my addr 172.21.15.88:0/2094333762 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-29T10:49:32.873 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:32.891+0000 7f350959a640 1 -- 172.21.15.88:0/2094333762 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2942566953 0 0) 0x7f350c0ae180 con 0x7f350c0aeca0 2024-03-29T10:49:32.873 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:32.891+0000 7f350959a640 10 monclient(hunting): _negotiate 2024-03-29T10:49:32.874 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:32.891+0000 7f350959a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:32.874 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:32.891+0000 7f350959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:32.874 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:32.891+0000 7f350959a640 10 monclient(hunting): my global_id is 4574 2024-03-29T10:49:32.874 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:32.891+0000 7f350959a640 1 -- 172.21.15.88:0/2094333762 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f34f4003520 con 0x7f350c0aeca0 2024-03-29T10:49:32.874 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:32.891+0000 7f350959a640 1 -- 172.21.15.88:0/2094333762 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2778537921 0 0) 0x7f34f4003520 con 0x7f350c0aeca0 2024-03-29T10:49:32.874 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:32.891+0000 7f350959a640 1 -- 172.21.15.88:0/2094333762 >> v1:172.21.15.196:6789/0 conn(0x7f350c0b08e0 legacy=0x7f350c13d3e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:49:32.875 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:32.893+0000 7f350959a640 1 -- 172.21.15.88:0/2094333762 >> v1:172.21.15.88:6789/0 conn(0x7f350c0b0500 legacy=0x7f350c13ff10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:49:32.878 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:32.893+0000 7f350959a640 10 monclient: _finish_hunting 0 2024-03-29T10:49:32.878 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:32.893+0000 7f350959a640 1 monclient: found mon.noname-b 2024-03-29T10:49:32.878 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:32.893+0000 7f350959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:49:32.878 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:32.893+0000 7f350959a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.88:6790/0 2024-03-29T10:49:32.878 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:32.893+0000 7f350959a640 1 -- 172.21.15.88:0/2094333762 --> v1:172.21.15.88:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f350c0accf0 con 0x7f350c0aeca0 2024-03-29T10:49:32.878 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:32.893+0000 7f350959a640 10 monclient: _finish_auth 0 2024-03-29T10:49:32.878 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:32.893+0000 7f350959a640 10 monclient: _check_auth_tickets 2024-03-29T10:49:32.878 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:32.893+0000 7f350959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:49:32.878 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:32.894+0000 7f35108e2640 5 monclient: authenticate success, global_id 4574 2024-03-29T10:49:32.879 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:32.894+0000 7f35108e2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:49:32.879 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:32.894+0000 7f350959a640 1 -- 172.21.15.88:0/2094333762 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f3504001a10 con 0x7f350c0aeca0 2024-03-29T10:49:32.879 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:32.894+0000 7f350959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:32.879 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:32.894+0000 7f350959a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:49:32.879 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:32.894+0000 7f350959a640 10 monclient: dump: 2024-03-29T10:49:32.879 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:49:32.879 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:49:32.879 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:49:32.879 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:49:32.879 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:32.879 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:49:32.879 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:49:32.879 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:49:32.879 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:49:32.879 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:49:32.880 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:32.894+0000 7f350959a640 1 -- 172.21.15.88:0/2094333762 <== mon.2 v1:172.21.15.88:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3504001c70 con 0x7f350c0aeca0 2024-03-29T10:49:32.880 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:32.894+0000 7f350959a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:49:32.880 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:32.894+0000 7f350959a640 1 -- 172.21.15.88:0/2094333762 <== mon.2 v1:172.21.15.88:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f35040020c0 con 0x7f350c0aeca0 2024-03-29T10:49:32.880 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:32.894+0000 7f350959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:32.880 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:32.894+0000 7f350959a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:49:32.880 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:32.894+0000 7f350959a640 10 monclient: dump: 2024-03-29T10:49:32.880 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:49:32.880 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:49:32.880 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:49:32.880 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:49:32.880 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:32.880 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:49:32.880 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:49:32.880 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:49:32.881 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:49:32.881 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:49:32.881 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:32.894+0000 7f35108e2640 10 monclient: get_monmap_and_config success 2024-03-29T10:49:32.881 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:32.894+0000 7f35108e2640 4 set_mon_vals no callback set 2024-03-29T10:49:32.881 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:32.895+0000 7f35108e2640 10 monclient: shutdown 2024-03-29T10:49:32.881 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:32.895+0000 7f35108e2640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:49:32.881 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:32.895+0000 7f35108e2640 1 -- 172.21.15.88:0/2094333762 >> v1:172.21.15.88:6790/0 conn(0x7f350c0aeca0 legacy=0x7f350c14b7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:32.881 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:32.895+0000 7f35108e2640 1 -- 172.21.15.88:0/2094333762 shutdown_connections 2024-03-29T10:49:32.881 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:32.895+0000 7f35108e2640 1 -- 172.21.15.88:0/2094333762 >> 172.21.15.88:0/2094333762 conn(0x7f350c06b660 msgr2=0x7f350c06ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:49:32.881 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:32.896+0000 7f35108e2640 1 -- 172.21.15.88:0/2094333762 shutdown_connections 2024-03-29T10:49:32.881 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:32.896+0000 7f35108e2640 1 -- 172.21.15.88:0/2094333762 wait complete. 2024-03-29T10:49:32.883 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:32.897+0000 7f35108e2640 10 monclient: build_initial_monmap 2024-03-29T10:49:32.883 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:32.897+0000 7f35108e2640 10 monclient: monmap: 2024-03-29T10:49:32.883 INFO:teuthology.misc.health.smithi088.stderr:epoch 0 2024-03-29T10:49:32.884 INFO:teuthology.misc.health.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:49:32.884 INFO:teuthology.misc.health.smithi088.stderr:last_changed 0.000000 2024-03-29T10:49:32.884 INFO:teuthology.misc.health.smithi088.stderr:created 0.000000 2024-03-29T10:49:32.884 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 0 (unknown) 2024-03-29T10:49:32.884 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:49:32.884 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-03-29T10:49:32.884 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-29T10:49:32.884 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-c 2024-03-29T10:49:32.884 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:49:32.885 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:32.897+0000 7f35108e2640 1 librados: starting msgr at 2024-03-29T10:49:32.885 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:32.897+0000 7f35108e2640 1 librados: starting objecter 2024-03-29T10:49:32.885 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:32.897+0000 7f35108e2640 1 Processor -- start 2024-03-29T10:49:32.885 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:32.898+0000 7f35108e2640 1 -- start start 2024-03-29T10:49:32.885 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:32.898+0000 7f35108e2640 1 librados: setting wanted keys 2024-03-29T10:49:32.885 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:32.898+0000 7f35108e2640 1 librados: calling monclient init 2024-03-29T10:49:32.885 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:32.898+0000 7f35108e2640 10 monclient: init 2024-03-29T10:49:32.885 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:32.898+0000 7f35108e2640 10 monclient: _reopen_session rank -1 2024-03-29T10:49:32.885 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:32.898+0000 7f35108e2640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:49:32.885 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:32.898+0000 7f35108e2640 10 monclient(hunting): picked mon.noname-b con 0x7f350c0aeca0 addr v1:172.21.15.196:6789/0 2024-03-29T10:49:32.885 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:32.899+0000 7f35108e2640 10 monclient(hunting): picked mon.noname-c con 0x7f350c0b0500 addr v1:172.21.15.88:6790/0 2024-03-29T10:49:32.885 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:32.899+0000 7f3509d9b640 1 --1- >> v1:172.21.15.88:6790/0 conn(0x7f350c0b0500 0x7f350c1dee50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6790/0 says I am v1:172.21.15.88:53152/0 (socket says 172.21.15.88:53152) 2024-03-29T10:49:32.885 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:32.899+0000 7f3509d9b640 1 -- 172.21.15.88:0/3143685890 learned_addr learned my addr 172.21.15.88:0/3143685890 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-29T10:49:32.886 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:32.899+0000 7f34faffd640 10 client.?.objecter ms_handle_connect 0x7f350c0b0500 2024-03-29T10:49:32.886 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:32.899+0000 7f34faffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:49:32.886 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:32.900+0000 7f35108e2640 10 monclient(hunting): picked mon.noname-a con 0x7f350c070660 addr v1:172.21.15.88:6789/0 2024-03-29T10:49:32.886 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:32.900+0000 7f35108e2640 1 -- 172.21.15.88:0/3143685890 --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f350c11efb0 con 0x7f350c070660 2024-03-29T10:49:32.886 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:32.900+0000 7f35108e2640 1 -- 172.21.15.88:0/3143685890 --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f350c13cfc0 con 0x7f350c0aeca0 2024-03-29T10:49:32.886 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:32.900+0000 7f35108e2640 1 -- 172.21.15.88:0/3143685890 --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f350c11e290 con 0x7f350c0b0500 2024-03-29T10:49:32.886 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:32.900+0000 7f35108e2640 10 monclient(hunting): _renew_subs 2024-03-29T10:49:32.886 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:32.900+0000 7f35108e2640 10 monclient(hunting): authenticate will time out at 1295.136211s 2024-03-29T10:49:32.886 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:32.900+0000 7f34faffd640 1 -- 172.21.15.88:0/3143685890 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3266322879 0 0) 0x7f350c11e290 con 0x7f350c0b0500 2024-03-29T10:49:32.887 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:32.900+0000 7f34faffd640 10 monclient(hunting): _negotiate 2024-03-29T10:49:32.887 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:32.900+0000 7f34faffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:32.887 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:32.900+0000 7f34faffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:32.887 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:32.900+0000 7f34faffd640 10 monclient(hunting): my global_id is 4577 2024-03-29T10:49:32.887 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:32.900+0000 7f34faffd640 1 -- 172.21.15.88:0/3143685890 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f34ec004eb0 con 0x7f350c0b0500 2024-03-29T10:49:32.887 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:32.901+0000 7f34faffd640 1 -- 172.21.15.88:0/3143685890 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2179569783 0 0) 0x7f34ec004eb0 con 0x7f350c0b0500 2024-03-29T10:49:32.887 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:32.901+0000 7f34faffd640 1 -- 172.21.15.88:0/3143685890 --> v1:172.21.15.88:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f350c11e290 con 0x7f350c0b0500 2024-03-29T10:49:32.887 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:32.901+0000 7f34faffd640 1 -- 172.21.15.88:0/3143685890 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f35000017d0 con 0x7f350c0b0500 2024-03-29T10:49:32.887 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:32.901+0000 7f34faffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:32.887 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:32.901+0000 7f34faffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:49:32.887 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:32.901+0000 7f34faffd640 10 monclient(hunting): dump: 2024-03-29T10:49:32.887 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:49:32.888 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:49:32.888 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:49:32.888 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:49:32.888 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:32.888 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:49:32.888 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:49:32.888 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:49:32.888 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:49:32.888 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:49:32.889 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:32.901+0000 7f34faffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:49:32.889 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:32.901+0000 7f34faffd640 1 -- 172.21.15.88:0/3143685890 <== mon.2 v1:172.21.15.88:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1177965800 0 0) 0x7f350c11e290 con 0x7f350c0b0500 2024-03-29T10:49:32.889 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:32.901+0000 7f34faffd640 1 -- 172.21.15.88:0/3143685890 >> v1:172.21.15.196:6789/0 conn(0x7f350c0aeca0 legacy=0x7f350c11d510 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:49:32.894 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:32.902+0000 7f34faffd640 1 -- 172.21.15.88:0/3143685890 >> v1:172.21.15.88:6789/0 conn(0x7f350c070660 legacy=0x7f350c13c270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:32.894 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:32.902+0000 7f34faffd640 10 monclient: _finish_hunting 0 2024-03-29T10:49:32.894 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:32.902+0000 7f34faffd640 1 monclient: found mon.c 2024-03-29T10:49:32.894 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:32.902+0000 7f34faffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:49:32.894 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:32.902+0000 7f34faffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-03-29T10:49:32.894 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:32.902+0000 7f34faffd640 1 -- 172.21.15.88:0/3143685890 --> v1:172.21.15.88:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f350c11dd70 con 0x7f350c0b0500 2024-03-29T10:49:32.894 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:32.902+0000 7f34faffd640 10 monclient: _finish_auth 0 2024-03-29T10:49:32.894 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:32.902+0000 7f34faffd640 10 monclient: _check_auth_tickets 2024-03-29T10:49:32.895 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:32.902+0000 7f34faffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:49:32.895 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:32.902+0000 7f34faffd640 10 client.?.objecter ms_handle_connect 0x7f350c070660 2024-03-29T10:49:32.895 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:32.902+0000 7f34faffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:49:32.895 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:32.902+0000 7f35108e2640 5 monclient: authenticate success, global_id 4577 2024-03-29T10:49:32.895 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:32.902+0000 7f35108e2640 10 monclient: _renew_subs 2024-03-29T10:49:32.895 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:32.903+0000 7f35108e2640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-03-29T10:49:32.895 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:32.903+0000 7f35108e2640 1 -- 172.21.15.88:0/3143685890 --> v1:172.21.15.88:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f350c1df560 con 0x7f350c0b0500 2024-03-29T10:49:32.895 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:32.903+0000 7f35108e2640 10 client.4577.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:49:32.895 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:32.903+0000 7f35108e2640 10 monclient: _renew_subs 2024-03-29T10:49:32.895 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:32.903+0000 7f35108e2640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-03-29T10:49:32.895 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:32.903+0000 7f35108e2640 1 -- 172.21.15.88:0/3143685890 --> v1:172.21.15.88:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f350c13c980 con 0x7f350c0b0500 2024-03-29T10:49:32.895 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:32.903+0000 7f35108e2640 1 librados: init done 2024-03-29T10:49:32.896 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:32.906+0000 7f34faffd640 1 -- 172.21.15.88:0/3143685890 <== mon.2 v1:172.21.15.88:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3500001930 con 0x7f350c0b0500 2024-03-29T10:49:32.896 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:32.906+0000 7f34faffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:49:32.896 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:32.906+0000 7f34faffd640 1 -- 172.21.15.88:0/3143685890 <== mon.2 v1:172.21.15.88:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f3500004a50 con 0x7f350c0b0500 2024-03-29T10:49:32.896 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:32.906+0000 7f34faffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:32.896 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:32.906+0000 7f34faffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:49:32.896 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:32.906+0000 7f35108e2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:49:32.896 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:32.906+0000 7f34faffd640 10 monclient: dump: 2024-03-29T10:49:32.896 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:49:32.896 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:49:32.897 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:49:32.897 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:49:32.897 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:32.897 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:49:32.897 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:49:32.897 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:49:32.897 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:49:32.897 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:49:32.897 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:32.906+0000 7f35108e2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:49:32.897 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:32.906+0000 7f35108e2640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-03-29T10:49:32.897 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:32.906+0000 7f35108e2640 1 -- 172.21.15.88:0/3143685890 --> v1:172.21.15.88:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f350c0ad0c0 con 0x7f350c0b0500 2024-03-29T10:49:32.897 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:32.907+0000 7f34faffd640 1 -- 172.21.15.88:0/3143685890 <== mon.2 v1:172.21.15.88:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4235629220 0 0) 0x7f350001aae0 con 0x7f350c0b0500 2024-03-29T10:49:32.897 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:32.907+0000 7f3508d99640 4 set_mon_vals no callback set 2024-03-29T10:49:32.897 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:32.907+0000 7f34faffd640 1 -- 172.21.15.88:0/3143685890 <== mon.2 v1:172.21.15.88:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4257673582 0 0) 0x7f3500084a70 con 0x7f350c0b0500 2024-03-29T10:49:32.897 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:32.907+0000 7f34faffd640 10 client.4577.objecter ms_dispatch 0x7f350c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:49:32.898 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:32.907+0000 7f34faffd640 3 client.4577.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:49:32.898 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:32.907+0000 7f34faffd640 3 client.4577.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:49:32.898 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:32.908+0000 7f34faffd640 20 client.4577.objecter dump_active .. 0 homeless 2024-03-29T10:49:32.898 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:32.908+0000 7f350a59c640 10 monclient: get_auth_request con 0x7f34ec06db30 auth_method 0 2024-03-29T10:49:32.898 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:32.908+0000 7f34faffd640 10 client.4577.objecter ms_handle_connect 0x7f34ec06db30 2024-03-29T10:49:32.898 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:32.915+0000 7f34faffd640 1 -- 172.21.15.88:0/3143685890 <== mon.2 v1:172.21.15.88:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f3500050180 con 0x7f350c0b0500 2024-03-29T10:49:32.898 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:32.915+0000 7f34faffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:49:32.898 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:32.915+0000 7f34faffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:49:33.220 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:33.238+0000 7f35108e2640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:49:33.221 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:33.238+0000 7f35108e2640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:49:33.221 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:33.238+0000 7f35108e2640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-03-29T10:49:33.221 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:33.238+0000 7f35108e2640 1 -- 172.21.15.88:0/3143685890 --> v1:172.21.15.88:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f350c070a90 con 0x7f350c0b0500 2024-03-29T10:49:33.221 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:33.239+0000 7f34faffd640 1 -- 172.21.15.88:0/3143685890 <== mon.2 v1:172.21.15.88:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 344506374) 0x7f350c070a90 con 0x7f350c0b0500 2024-03-29T10:49:33.221 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:33.239+0000 7f34faffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:49:33.221 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:33.239+0000 7f34faffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:49:33.222 INFO:teuthology.misc.health.smithi088.stdout: 2024-03-29T10:49:33.222 INFO:teuthology.misc.health.smithi088.stdout:{"epoch":10,"fsid":"869f5188-f92b-42dd-a46d-8c646e0388a9","created":"2024-03-29T10:40:40.285087+0000","modified":"2024-03-29T10:40:42.536230+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:40:42.536230+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a26c0c1c-8de0-438c-a3f6-3df10ea61574","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5de24915-8dec-49b6-8b16-f3170197c591","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ac054af3-d589-4565-9644-e6af75975c53","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3ec4f045-c714-4a22-bd1a-e07330b45ab3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"cddb9867-8e30-4951-9b13-3d1e78b9c45a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dc6d9911-785d-432f-8b53-56a1ccab302f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"77c88437-b587-496e-8b4d-1824ea8b5305","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ce963d96-5a8e-44c3-99de-c0dbebe234ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:49:33.224 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:33.242+0000 7f35108e2640 10 librados: watch_flush enter 2024-03-29T10:49:33.227 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:33.245+0000 7f35108e2640 10 librados: watch_flush exit 2024-03-29T10:49:33.227 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:33.245+0000 7f35108e2640 20 client.4577.objecter shutdown clearing up homeless session... 2024-03-29T10:49:33.227 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:33.245+0000 7f35108e2640 10 client.4577.objecter successfully canceled tick 2024-03-29T10:49:33.227 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:33.245+0000 7f35108e2640 1 -- 172.21.15.88:0/3143685890 >> v1:172.21.15.88:6816/1752861349 conn(0x7f34ec06db30 legacy=0x7f34ec06fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:33.227 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:33.245+0000 7f35108e2640 10 monclient: shutdown 2024-03-29T10:49:33.227 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:33.245+0000 7f35108e2640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:49:33.227 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:33.245+0000 7f35108e2640 1 -- 172.21.15.88:0/3143685890 >> v1:172.21.15.88:6790/0 conn(0x7f350c0b0500 legacy=0x7f350c1dee50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:33.227 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:33.245+0000 7f35108e2640 1 -- 172.21.15.88:0/3143685890 shutdown_connections 2024-03-29T10:49:33.228 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:33.245+0000 7f35108e2640 1 -- 172.21.15.88:0/3143685890 >> 172.21.15.88:0/3143685890 conn(0x7f350c06b660 msgr2=0x7f350c0a9c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:49:33.228 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:33.245+0000 7f35108e2640 1 -- 172.21.15.88:0/3143685890 shutdown_connections 2024-03-29T10:49:33.228 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:33.245+0000 7f35108e2640 1 -- 172.21.15.88:0/3143685890 wait complete. 2024-03-29T10:49:33.228 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:33.245+0000 7f35108e2640 1 librados: shutdown 2024-03-29T10:49:33.244 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:49:40.052 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:49:40.197 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:40.215+0000 7ffa1a27b640 10 monclient: get_monmap_and_config 2024-03-29T10:49:40.197 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:40.215+0000 7ffa1a27b640 10 monclient: build_initial_monmap 2024-03-29T10:49:40.197 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:40.215+0000 7ffa1a27b640 10 monclient: monmap: 2024-03-29T10:49:40.198 INFO:teuthology.misc.health.smithi088.stderr:epoch 0 2024-03-29T10:49:40.198 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:49:40.198 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:49:40.216461+0000 2024-03-29T10:49:40.198 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:49:40.216461+0000 2024-03-29T10:49:40.198 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 0 (unknown) 2024-03-29T10:49:40.198 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:49:40.198 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-03-29T10:49:40.198 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-29T10:49:40.198 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-b 2024-03-29T10:49:40.198 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:49:40.199 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:40.216+0000 7ffa1a27b640 1 Processor -- start 2024-03-29T10:49:40.199 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:40.216+0000 7ffa1a27b640 1 -- start start 2024-03-29T10:49:40.199 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:40.216+0000 7ffa1a27b640 10 monclient: init 2024-03-29T10:49:40.199 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:40.217+0000 7ffa1a27b640 10 monclient: _reopen_session rank -1 2024-03-29T10:49:40.199 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:40.217+0000 7ffa1a27b640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:49:40.201 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:40.217+0000 7ffa1a27b640 10 monclient(hunting): picked mon.noname-b con 0x7ffa140b0860 addr v1:172.21.15.88:6790/0 2024-03-29T10:49:40.201 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:40.217+0000 7ffa1a27b640 10 monclient(hunting): picked mon.noname-c con 0x7ffa1413ef30 addr v1:172.21.15.196:6789/0 2024-03-29T10:49:40.201 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:40.218+0000 7ffa1a27b640 10 monclient(hunting): picked mon.noname-a con 0x7ffa140af8e0 addr v1:172.21.15.88:6789/0 2024-03-29T10:49:40.201 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:40.218+0000 7ffa1a27b640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa140b0c40 con 0x7ffa140af8e0 2024-03-29T10:49:40.201 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:40.218+0000 7ffa1a27b640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa140aed10 con 0x7ffa1413ef30 2024-03-29T10:49:40.202 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:40.218+0000 7ffa1a27b640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa140afd50 con 0x7ffa140b0860 2024-03-29T10:49:40.202 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:40.218+0000 7ffa1a27b640 10 monclient(hunting): _renew_subs 2024-03-29T10:49:40.202 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:40.218+0000 7ffa1a27b640 10 monclient(hunting): authenticate will time out at 1302.454192s 2024-03-29T10:49:40.202 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:40.219+0000 7ffa13fff640 1 --1- >> v1:172.21.15.88:6790/0 conn(0x7ffa140b0860 0x7ffa1413e820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6790/0 says I am v1:172.21.15.88:52442/0 (socket says 172.21.15.88:52442) 2024-03-29T10:49:40.202 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:40.219+0000 7ffa13fff640 1 -- 172.21.15.88:0/946927149 learned_addr learned my addr 172.21.15.88:0/946927149 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-29T10:49:40.211 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:40.225+0000 7ffa137fe640 1 -- 172.21.15.88:0/946927149 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4243008670 0 0) 0x7ffa140b0c40 con 0x7ffa140af8e0 2024-03-29T10:49:40.211 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:40.225+0000 7ffa137fe640 10 monclient(hunting): _negotiate 2024-03-29T10:49:40.211 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:40.225+0000 7ffa137fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:40.211 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:40.225+0000 7ffa137fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:40.211 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:40.225+0000 7ffa137fe640 10 monclient(hunting): my global_id is 4554 2024-03-29T10:49:40.211 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:40.225+0000 7ffa137fe640 1 -- 172.21.15.88:0/946927149 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa00003520 con 0x7ffa140af8e0 2024-03-29T10:49:40.211 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:40.226+0000 7ffa137fe640 1 -- 172.21.15.88:0/946927149 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3396658986 0 0) 0x7ffa140afd50 con 0x7ffa140b0860 2024-03-29T10:49:40.211 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:40.226+0000 7ffa137fe640 10 monclient(hunting): _negotiate 2024-03-29T10:49:40.211 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:40.226+0000 7ffa137fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:40.211 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:40.226+0000 7ffa137fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:40.211 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:40.226+0000 7ffa137fe640 10 monclient(hunting): my global_id is 4580 2024-03-29T10:49:40.211 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:40.226+0000 7ffa137fe640 1 -- 172.21.15.88:0/946927149 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa140b0c40 con 0x7ffa140b0860 2024-03-29T10:49:40.212 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:40.226+0000 7ffa137fe640 1 -- 172.21.15.88:0/946927149 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 567041586 0 0) 0x7ffa140aed10 con 0x7ffa1413ef30 2024-03-29T10:49:40.212 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:40.226+0000 7ffa137fe640 10 monclient(hunting): _negotiate 2024-03-29T10:49:40.212 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:40.226+0000 7ffa137fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:40.212 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:40.226+0000 7ffa137fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:40.212 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:40.226+0000 7ffa137fe640 10 monclient(hunting): my global_id is 4525 2024-03-29T10:49:40.212 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:40.226+0000 7ffa137fe640 1 -- 172.21.15.88:0/946927149 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa140afd50 con 0x7ffa1413ef30 2024-03-29T10:49:40.212 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:40.227+0000 7ffa137fe640 1 -- 172.21.15.88:0/946927149 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1484042800 0 0) 0x7ffa00003520 con 0x7ffa140af8e0 2024-03-29T10:49:40.212 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:40.227+0000 7ffa137fe640 1 -- 172.21.15.88:0/946927149 >> v1:172.21.15.88:6790/0 conn(0x7ffa140b0860 legacy=0x7ffa1413e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:40.212 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:40.227+0000 7ffa137fe640 1 -- 172.21.15.88:0/946927149 >> v1:172.21.15.196:6789/0 conn(0x7ffa1413ef30 legacy=0x7ffa140af1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:40.212 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:40.227+0000 7ffa137fe640 10 monclient: _finish_hunting 0 2024-03-29T10:49:40.212 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:40.227+0000 7ffa137fe640 1 monclient: found mon.noname-a 2024-03-29T10:49:40.212 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:40.227+0000 7ffa137fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:49:40.212 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:40.227+0000 7ffa137fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.88:6789/0 2024-03-29T10:49:40.213 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:40.227+0000 7ffa137fe640 1 -- 172.21.15.88:0/946927149 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa140ab1b0 con 0x7ffa140af8e0 2024-03-29T10:49:40.213 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:40.227+0000 7ffa137fe640 10 monclient: _finish_auth 0 2024-03-29T10:49:40.213 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:40.227+0000 7ffa137fe640 10 monclient: _check_auth_tickets 2024-03-29T10:49:40.213 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:40.227+0000 7ffa137fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:49:40.213 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:40.227+0000 7ffa137fe640 1 -- 172.21.15.88:0/946927149 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7ffa0c001990 con 0x7ffa140af8e0 2024-03-29T10:49:40.213 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:40.228+0000 7ffa137fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:40.213 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:40.228+0000 7ffa137fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:49:40.213 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:40.228+0000 7ffa137fe640 10 monclient: dump: 2024-03-29T10:49:40.213 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:49:40.213 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:49:40.213 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:49:40.213 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:49:40.213 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:40.214 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:49:40.214 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:49:40.214 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:49:40.214 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:49:40.214 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:49:40.214 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:40.228+0000 7ffa137fe640 1 -- 172.21.15.88:0/946927149 <== mon.0 v1:172.21.15.88:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ffa0c001bb0 con 0x7ffa140af8e0 2024-03-29T10:49:40.214 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:40.228+0000 7ffa137fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:49:40.214 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:40.228+0000 7ffa137fe640 1 -- 172.21.15.88:0/946927149 <== mon.0 v1:172.21.15.88:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7ffa0c001f40 con 0x7ffa140af8e0 2024-03-29T10:49:40.214 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:40.228+0000 7ffa137fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:40.214 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:40.228+0000 7ffa137fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:49:40.214 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:40.228+0000 7ffa137fe640 10 monclient: dump: 2024-03-29T10:49:40.214 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:49:40.214 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:49:40.214 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:49:40.214 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:49:40.215 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:40.215 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:49:40.215 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:49:40.215 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:49:40.215 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:49:40.215 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:49:40.219 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:40.228+0000 7ffa1a27b640 5 monclient: authenticate success, global_id 4554 2024-03-29T10:49:40.219 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:40.228+0000 7ffa1a27b640 10 monclient: get_monmap_and_config success 2024-03-29T10:49:40.219 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:40.228+0000 7ffa1a27b640 4 set_mon_vals no callback set 2024-03-29T10:49:40.219 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:40.229+0000 7ffa1a27b640 10 monclient: shutdown 2024-03-29T10:49:40.219 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:40.229+0000 7ffa1a27b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:49:40.219 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:40.229+0000 7ffa1a27b640 1 -- 172.21.15.88:0/946927149 >> v1:172.21.15.88:6789/0 conn(0x7ffa140af8e0 legacy=0x7ffa14149a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:40.219 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:40.231+0000 7ffa1a27b640 1 -- 172.21.15.88:0/946927149 shutdown_connections 2024-03-29T10:49:40.220 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:40.231+0000 7ffa1a27b640 1 -- 172.21.15.88:0/946927149 >> 172.21.15.88:0/946927149 conn(0x7ffa1406b660 msgr2=0x7ffa1406ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:49:40.220 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:40.231+0000 7ffa1a27b640 1 -- 172.21.15.88:0/946927149 shutdown_connections 2024-03-29T10:49:40.220 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:40.231+0000 7ffa1a27b640 1 -- 172.21.15.88:0/946927149 wait complete. 2024-03-29T10:49:40.220 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:40.232+0000 7ffa1a27b640 10 monclient: build_initial_monmap 2024-03-29T10:49:40.220 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:40.232+0000 7ffa1a27b640 10 monclient: monmap: 2024-03-29T10:49:40.220 INFO:teuthology.misc.health.smithi088.stderr:epoch 0 2024-03-29T10:49:40.220 INFO:teuthology.misc.health.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:49:40.220 INFO:teuthology.misc.health.smithi088.stderr:last_changed 0.000000 2024-03-29T10:49:40.220 INFO:teuthology.misc.health.smithi088.stderr:created 0.000000 2024-03-29T10:49:40.220 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 0 (unknown) 2024-03-29T10:49:40.220 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:49:40.220 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-03-29T10:49:40.221 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-29T10:49:40.221 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-c 2024-03-29T10:49:40.221 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:49:40.221 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:40.233+0000 7ffa1a27b640 1 librados: starting msgr at 2024-03-29T10:49:40.221 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:40.233+0000 7ffa1a27b640 1 librados: starting objecter 2024-03-29T10:49:40.221 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:40.233+0000 7ffa1a27b640 1 Processor -- start 2024-03-29T10:49:40.221 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:40.233+0000 7ffa1a27b640 1 -- start start 2024-03-29T10:49:40.221 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:40.233+0000 7ffa1a27b640 1 librados: setting wanted keys 2024-03-29T10:49:40.221 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:40.233+0000 7ffa1a27b640 1 librados: calling monclient init 2024-03-29T10:49:40.221 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:40.233+0000 7ffa1a27b640 10 monclient: init 2024-03-29T10:49:40.221 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:40.234+0000 7ffa1a27b640 10 monclient: _reopen_session rank -1 2024-03-29T10:49:40.221 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:40.234+0000 7ffa1a27b640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:49:40.221 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:40.234+0000 7ffa1a27b640 10 monclient(hunting): picked mon.noname-c con 0x7ffa140af8e0 addr v1:172.21.15.88:6790/0 2024-03-29T10:49:40.221 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:40.234+0000 7ffa13fff640 1 --1- >> v1:172.21.15.88:6790/0 conn(0x7ffa140af8e0 0x7ffa140b3b50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6790/0 says I am v1:172.21.15.88:53432/0 (socket says 172.21.15.88:53432) 2024-03-29T10:49:40.221 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:40.234+0000 7ffa13fff640 1 -- 172.21.15.88:0/3682765528 learned_addr learned my addr 172.21.15.88:0/3682765528 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-29T10:49:40.222 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:40.234+0000 7ffa1a27b640 10 monclient(hunting): picked mon.noname-b con 0x7ffa140b0860 addr v1:172.21.15.196:6789/0 2024-03-29T10:49:40.222 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:40.234+0000 7ffa1a27b640 10 monclient(hunting): picked mon.noname-a con 0x7ffa1413ef30 addr v1:172.21.15.88:6789/0 2024-03-29T10:49:40.222 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:40.235+0000 7ffa1a27b640 1 -- 172.21.15.88:0/3682765528 --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa140b3170 con 0x7ffa1413ef30 2024-03-29T10:49:40.222 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:40.235+0000 7ffa1a27b640 1 -- 172.21.15.88:0/3682765528 --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa14135580 con 0x7ffa140b0860 2024-03-29T10:49:40.222 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:40.235+0000 7ffa1a27b640 1 -- 172.21.15.88:0/3682765528 --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa14135ed0 con 0x7ffa140af8e0 2024-03-29T10:49:40.222 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:40.235+0000 7ffa1a27b640 10 monclient(hunting): _renew_subs 2024-03-29T10:49:40.222 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:40.235+0000 7ffa1a27b640 10 monclient(hunting): authenticate will time out at 1302.471262s 2024-03-29T10:49:40.222 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:40.235+0000 7ffa117fa640 10 client.?.objecter ms_handle_connect 0x7ffa140af8e0 2024-03-29T10:49:40.222 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:40.235+0000 7ffa117fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:49:40.222 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:40.237+0000 7ffa117fa640 1 -- 172.21.15.88:0/3682765528 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4123959800 0 0) 0x7ffa14135ed0 con 0x7ffa140af8e0 2024-03-29T10:49:40.222 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:40.237+0000 7ffa117fa640 10 monclient(hunting): _negotiate 2024-03-29T10:49:40.222 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:40.237+0000 7ffa117fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:40.222 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:40.237+0000 7ffa117fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:40.222 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:40.237+0000 7ffa117fa640 10 monclient(hunting): my global_id is 4583 2024-03-29T10:49:40.223 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:40.237+0000 7ffa117fa640 1 -- 172.21.15.88:0/3682765528 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff9f4004eb0 con 0x7ffa140af8e0 2024-03-29T10:49:40.227 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:40.241+0000 7ffa117fa640 10 client.?.objecter ms_handle_connect 0x7ffa140b0860 2024-03-29T10:49:40.227 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:40.241+0000 7ffa117fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:49:40.227 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:40.241+0000 7ffa117fa640 10 client.?.objecter ms_handle_connect 0x7ffa1413ef30 2024-03-29T10:49:40.227 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:40.241+0000 7ffa117fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:49:40.227 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:40.241+0000 7ffa117fa640 1 -- 172.21.15.88:0/3682765528 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 754422792 0 0) 0x7ff9f4004eb0 con 0x7ffa140af8e0 2024-03-29T10:49:40.227 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:40.241+0000 7ffa117fa640 1 -- 172.21.15.88:0/3682765528 --> v1:172.21.15.88:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ffa14135ed0 con 0x7ffa140af8e0 2024-03-29T10:49:40.227 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:40.242+0000 7ffa117fa640 1 -- 172.21.15.88:0/3682765528 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3451019317 0 0) 0x7ffa14135580 con 0x7ffa140b0860 2024-03-29T10:49:40.227 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:40.242+0000 7ffa117fa640 10 monclient(hunting): _negotiate 2024-03-29T10:49:40.227 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:40.242+0000 7ffa117fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:40.227 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:40.242+0000 7ffa117fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:40.227 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:40.242+0000 7ffa117fa640 10 monclient(hunting): my global_id is 4528 2024-03-29T10:49:40.227 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:40.242+0000 7ffa117fa640 1 -- 172.21.15.88:0/3682765528 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff9f4004eb0 con 0x7ffa140b0860 2024-03-29T10:49:40.228 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:40.242+0000 7ffa117fa640 1 -- 172.21.15.88:0/3682765528 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 179709394 0 0) 0x7ffa140b3170 con 0x7ffa1413ef30 2024-03-29T10:49:40.228 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:40.242+0000 7ffa117fa640 10 monclient(hunting): _negotiate 2024-03-29T10:49:40.228 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:40.242+0000 7ffa117fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:40.228 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:40.242+0000 7ffa117fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:40.228 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:40.242+0000 7ffa117fa640 10 monclient(hunting): my global_id is 4557 2024-03-29T10:49:40.228 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:40.242+0000 7ffa117fa640 1 -- 172.21.15.88:0/3682765528 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa14135580 con 0x7ffa1413ef30 2024-03-29T10:49:40.228 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:40.242+0000 7ffa117fa640 1 -- 172.21.15.88:0/3682765528 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7ffa04001a50 con 0x7ffa140af8e0 2024-03-29T10:49:40.228 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:40.243+0000 7ffa117fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:40.228 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:40.243+0000 7ffa117fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:49:40.228 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:40.243+0000 7ffa117fa640 10 monclient(hunting): dump: 2024-03-29T10:49:40.228 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:49:40.228 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:49:40.228 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:49:40.228 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:49:40.228 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:40.229 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:49:40.229 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:49:40.229 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:49:40.229 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:49:40.229 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:49:40.229 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:40.243+0000 7ffa117fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:49:40.229 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:40.243+0000 7ffa117fa640 1 -- 172.21.15.88:0/3682765528 <== mon.2 v1:172.21.15.88:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3452527821 0 0) 0x7ffa14135ed0 con 0x7ffa140af8e0 2024-03-29T10:49:40.229 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:40.243+0000 7ffa117fa640 1 -- 172.21.15.88:0/3682765528 >> v1:172.21.15.196:6789/0 conn(0x7ffa140b0860 legacy=0x7ffa141ebab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:40.229 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:40.243+0000 7ffa117fa640 1 -- 172.21.15.88:0/3682765528 >> v1:172.21.15.88:6789/0 conn(0x7ffa1413ef30 legacy=0x7ffa141ef1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:40.229 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:40.243+0000 7ffa117fa640 10 monclient: _finish_hunting 0 2024-03-29T10:49:40.229 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:40.243+0000 7ffa117fa640 1 monclient: found mon.c 2024-03-29T10:49:40.229 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:40.243+0000 7ffa117fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:49:40.229 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:40.243+0000 7ffa117fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-03-29T10:49:40.229 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:40.243+0000 7ffa117fa640 1 -- 172.21.15.88:0/3682765528 --> v1:172.21.15.88:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa141361b0 con 0x7ffa140af8e0 2024-03-29T10:49:40.230 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:40.243+0000 7ffa117fa640 10 monclient: _finish_auth 0 2024-03-29T10:49:40.230 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:40.243+0000 7ffa117fa640 10 monclient: _check_auth_tickets 2024-03-29T10:49:40.230 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:40.243+0000 7ffa117fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:49:40.230 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:40.243+0000 7ffa1a27b640 5 monclient: authenticate success, global_id 4583 2024-03-29T10:49:40.230 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:40.244+0000 7ffa1a27b640 10 monclient: _renew_subs 2024-03-29T10:49:40.230 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:40.244+0000 7ffa1a27b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-03-29T10:49:40.230 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:40.244+0000 7ffa1a27b640 1 -- 172.21.15.88:0/3682765528 --> v1:172.21.15.88:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa14136490 con 0x7ffa140af8e0 2024-03-29T10:49:40.230 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:40.244+0000 7ffa1a27b640 10 client.4583.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:49:40.230 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:40.244+0000 7ffa1a27b640 10 monclient: _renew_subs 2024-03-29T10:49:40.230 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:40.244+0000 7ffa1a27b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-03-29T10:49:40.230 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:40.244+0000 7ffa1a27b640 1 -- 172.21.15.88:0/3682765528 --> v1:172.21.15.88:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa1413b740 con 0x7ffa140af8e0 2024-03-29T10:49:40.230 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:40.244+0000 7ffa1a27b640 1 librados: init done 2024-03-29T10:49:40.232 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:40.245+0000 7ffa1a27b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:49:40.232 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:40.245+0000 7ffa1a27b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:49:40.232 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:40.246+0000 7ffa1a27b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-03-29T10:49:40.232 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:40.246+0000 7ffa1a27b640 1 -- 172.21.15.88:0/3682765528 --> v1:172.21.15.88:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa14135710 con 0x7ffa140af8e0 2024-03-29T10:49:40.232 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:40.246+0000 7ffa117fa640 1 -- 172.21.15.88:0/3682765528 <== mon.2 v1:172.21.15.88:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ffa04001d70 con 0x7ffa140af8e0 2024-03-29T10:49:40.232 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:40.246+0000 7ffa117fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:49:40.232 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:40.246+0000 7ffa117fa640 1 -- 172.21.15.88:0/3682765528 <== mon.2 v1:172.21.15.88:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7ffa04001bf0 con 0x7ffa140af8e0 2024-03-29T10:49:40.232 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:40.246+0000 7ffa117fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:40.232 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:40.246+0000 7ffa117fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:49:40.232 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:40.246+0000 7ffa117fa640 10 monclient: dump: 2024-03-29T10:49:40.233 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:49:40.233 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:49:40.233 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:49:40.233 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:49:40.233 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:40.233 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:49:40.233 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:49:40.233 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:49:40.233 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:49:40.233 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:49:40.233 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:40.246+0000 7ffa12ffd640 4 set_mon_vals no callback set 2024-03-29T10:49:40.233 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:40.247+0000 7ffa117fa640 1 -- 172.21.15.88:0/3682765528 <== mon.2 v1:172.21.15.88:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4235629220 0 0) 0x7ffa0401ab60 con 0x7ffa140af8e0 2024-03-29T10:49:40.233 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:40.248+0000 7ffa117fa640 1 -- 172.21.15.88:0/3682765528 <== mon.2 v1:172.21.15.88:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4257673582 0 0) 0x7ffa04084a60 con 0x7ffa140af8e0 2024-03-29T10:49:40.233 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:40.248+0000 7ffa117fa640 10 client.4583.objecter ms_dispatch 0x7ffa14000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:49:40.234 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:40.248+0000 7ffa117fa640 3 client.4583.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:49:40.234 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:40.248+0000 7ffa0bfff640 10 monclient: get_auth_request con 0x7ff9f406de00 auth_method 0 2024-03-29T10:49:40.234 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:40.248+0000 7ffa117fa640 3 client.4583.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:49:40.234 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:40.248+0000 7ffa117fa640 20 client.4583.objecter dump_active .. 0 homeless 2024-03-29T10:49:40.234 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:40.248+0000 7ffa117fa640 10 client.4583.objecter ms_handle_connect 0x7ff9f406de00 2024-03-29T10:49:40.237 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:40.255+0000 7ffa117fa640 1 -- 172.21.15.88:0/3682765528 <== mon.2 v1:172.21.15.88:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ffa04050170 con 0x7ffa140af8e0 2024-03-29T10:49:40.238 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:40.255+0000 7ffa117fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:49:40.238 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:40.255+0000 7ffa117fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:49:40.562 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:40.580+0000 7ffa1a27b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:49:40.562 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:40.580+0000 7ffa1a27b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:49:40.562 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:40.580+0000 7ffa1a27b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-03-29T10:49:40.562 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:40.580+0000 7ffa1a27b640 1 -- 172.21.15.88:0/3682765528 --> v1:172.21.15.88:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ffa14075f10 con 0x7ffa140af8e0 2024-03-29T10:49:40.564 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:40.582+0000 7ffa117fa640 1 -- 172.21.15.88:0/3682765528 <== mon.2 v1:172.21.15.88:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 344506374) 0x7ffa04052270 con 0x7ffa140af8e0 2024-03-29T10:49:40.564 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:40.582+0000 7ffa117fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:49:40.564 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:40.582+0000 7ffa117fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:49:40.565 INFO:teuthology.misc.health.smithi088.stdout: 2024-03-29T10:49:40.566 INFO:teuthology.misc.health.smithi088.stdout:{"epoch":10,"fsid":"869f5188-f92b-42dd-a46d-8c646e0388a9","created":"2024-03-29T10:40:40.285087+0000","modified":"2024-03-29T10:40:42.536230+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:40:42.536230+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a26c0c1c-8de0-438c-a3f6-3df10ea61574","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5de24915-8dec-49b6-8b16-f3170197c591","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ac054af3-d589-4565-9644-e6af75975c53","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3ec4f045-c714-4a22-bd1a-e07330b45ab3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"cddb9867-8e30-4951-9b13-3d1e78b9c45a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dc6d9911-785d-432f-8b53-56a1ccab302f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"77c88437-b587-496e-8b4d-1824ea8b5305","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ce963d96-5a8e-44c3-99de-c0dbebe234ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:49:40.568 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:40.586+0000 7ffa1a27b640 10 librados: watch_flush enter 2024-03-29T10:49:40.572 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:40.589+0000 7ffa1a27b640 10 librados: watch_flush exit 2024-03-29T10:49:40.572 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:40.589+0000 7ffa1a27b640 20 client.4583.objecter shutdown clearing up homeless session... 2024-03-29T10:49:40.572 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:40.589+0000 7ffa1a27b640 10 client.4583.objecter successfully canceled tick 2024-03-29T10:49:40.572 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:40.590+0000 7ffa1a27b640 1 -- 172.21.15.88:0/3682765528 >> v1:172.21.15.88:6816/1752861349 conn(0x7ff9f406de00 legacy=0x7ff9f40702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:40.572 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:40.590+0000 7ffa1a27b640 10 monclient: shutdown 2024-03-29T10:49:40.572 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:40.590+0000 7ffa1a27b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:49:40.573 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:40.590+0000 7ffa1a27b640 1 -- 172.21.15.88:0/3682765528 >> v1:172.21.15.88:6790/0 conn(0x7ffa140af8e0 legacy=0x7ffa140b3b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:40.573 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:40.590+0000 7ffa1a27b640 1 -- 172.21.15.88:0/3682765528 shutdown_connections 2024-03-29T10:49:40.573 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:40.590+0000 7ffa1a27b640 1 -- 172.21.15.88:0/3682765528 >> 172.21.15.88:0/3682765528 conn(0x7ffa1406b660 msgr2=0x7ffa1413e5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:49:40.573 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:40.590+0000 7ffa1a27b640 1 -- 172.21.15.88:0/3682765528 shutdown_connections 2024-03-29T10:49:40.573 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:40.590+0000 7ffa1a27b640 1 -- 172.21.15.88:0/3682765528 wait complete. 2024-03-29T10:49:40.573 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:40.591+0000 7ffa1a27b640 1 librados: shutdown 2024-03-29T10:49:40.590 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:49:47.396 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:49:47.541 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:47.559+0000 7f5920354640 10 monclient: get_monmap_and_config 2024-03-29T10:49:47.542 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:47.559+0000 7f5920354640 10 monclient: build_initial_monmap 2024-03-29T10:49:47.542 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:47.559+0000 7f5920354640 10 monclient: monmap: 2024-03-29T10:49:47.542 INFO:teuthology.misc.health.smithi088.stderr:epoch 0 2024-03-29T10:49:47.542 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:49:47.542 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:49:47.560687+0000 2024-03-29T10:49:47.542 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:49:47.560687+0000 2024-03-29T10:49:47.542 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 0 (unknown) 2024-03-29T10:49:47.542 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:49:47.542 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-03-29T10:49:47.542 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-29T10:49:47.542 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-b 2024-03-29T10:49:47.542 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:49:47.544 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:47.561+0000 7f5920354640 1 Processor -- start 2024-03-29T10:49:47.544 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:47.561+0000 7f5920354640 1 -- start start 2024-03-29T10:49:47.545 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:47.561+0000 7f5920354640 10 monclient: init 2024-03-29T10:49:47.545 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:47.562+0000 7f5920354640 10 monclient: _reopen_session rank -1 2024-03-29T10:49:47.545 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:47.562+0000 7f5920354640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:49:47.545 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:47.562+0000 7f5920354640 10 monclient(hunting): picked mon.noname-b con 0x7f59180b0860 addr v1:172.21.15.88:6790/0 2024-03-29T10:49:47.545 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:47.562+0000 7f5920354640 10 monclient(hunting): picked mon.noname-c con 0x7f591813ef30 addr v1:172.21.15.196:6789/0 2024-03-29T10:49:47.545 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:47.562+0000 7f5920354640 10 monclient(hunting): picked mon.noname-a con 0x7f59180af8e0 addr v1:172.21.15.88:6789/0 2024-03-29T10:49:47.545 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:47.562+0000 7f5920354640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59180b0c40 con 0x7f59180af8e0 2024-03-29T10:49:47.545 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:47.562+0000 7f5920354640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59180aed10 con 0x7f591813ef30 2024-03-29T10:49:47.545 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:47.563+0000 7f5920354640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59180afd50 con 0x7f59180b0860 2024-03-29T10:49:47.545 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:47.563+0000 7f5920354640 10 monclient(hunting): _renew_subs 2024-03-29T10:49:47.545 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:47.563+0000 7f5920354640 10 monclient(hunting): authenticate will time out at 1309.798904s 2024-03-29T10:49:47.548 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:47.566+0000 7f591e0f0640 1 --1- >> v1:172.21.15.88:6790/0 conn(0x7f59180b0860 0x7f591813e820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6790/0 says I am v1:172.21.15.88:54691/0 (socket says 172.21.15.88:54691) 2024-03-29T10:49:47.548 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:47.566+0000 7f591e0f0640 1 -- 172.21.15.88:0/2511225152 learned_addr learned my addr 172.21.15.88:0/2511225152 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-29T10:49:47.552 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:47.567+0000 7f591d0ee640 1 -- 172.21.15.88:0/2511225152 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2781791471 0 0) 0x7f59180afd50 con 0x7f59180b0860 2024-03-29T10:49:47.552 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:47.567+0000 7f591d0ee640 10 monclient(hunting): _negotiate 2024-03-29T10:49:47.552 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:47.567+0000 7f591d0ee640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:47.552 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:47.567+0000 7f591d0ee640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:47.552 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:47.567+0000 7f591d0ee640 10 monclient(hunting): my global_id is 4586 2024-03-29T10:49:47.552 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:47.567+0000 7f591d0ee640 1 -- 172.21.15.88:0/2511225152 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5904003520 con 0x7f59180b0860 2024-03-29T10:49:47.553 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:47.570+0000 7f591d0ee640 1 -- 172.21.15.88:0/2511225152 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2884150027 0 0) 0x7f59180aed10 con 0x7f591813ef30 2024-03-29T10:49:47.553 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:47.570+0000 7f591d0ee640 10 monclient(hunting): _negotiate 2024-03-29T10:49:47.553 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:47.570+0000 7f591d0ee640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:47.553 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:47.570+0000 7f591d0ee640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:47.553 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:47.570+0000 7f591d0ee640 10 monclient(hunting): my global_id is 4531 2024-03-29T10:49:47.554 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:47.570+0000 7f591d0ee640 1 -- 172.21.15.88:0/2511225152 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f59180afd50 con 0x7f591813ef30 2024-03-29T10:49:47.554 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:47.571+0000 7f591d0ee640 1 -- 172.21.15.88:0/2511225152 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3674269199 0 0) 0x7f5904003520 con 0x7f59180b0860 2024-03-29T10:49:47.554 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:47.571+0000 7f591d0ee640 1 -- 172.21.15.88:0/2511225152 >> v1:172.21.15.196:6789/0 conn(0x7f591813ef30 legacy=0x7f59180af1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:47.554 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:47.571+0000 7f591d0ee640 1 -- 172.21.15.88:0/2511225152 >> v1:172.21.15.88:6789/0 conn(0x7f59180af8e0 legacy=0x7f5918149a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:47.554 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:47.571+0000 7f591d0ee640 10 monclient: _finish_hunting 0 2024-03-29T10:49:47.554 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:47.571+0000 7f591d0ee640 1 monclient: found mon.noname-b 2024-03-29T10:49:47.554 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:47.571+0000 7f591d0ee640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:49:47.554 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:47.571+0000 7f591d0ee640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.88:6790/0 2024-03-29T10:49:47.554 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:47.571+0000 7f591d0ee640 1 -- 172.21.15.88:0/2511225152 --> v1:172.21.15.88:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59180ab1b0 con 0x7f59180b0860 2024-03-29T10:49:47.554 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:47.571+0000 7f591d0ee640 10 monclient: _finish_auth 0 2024-03-29T10:49:47.554 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:47.571+0000 7f591d0ee640 10 monclient: _check_auth_tickets 2024-03-29T10:49:47.554 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:47.571+0000 7f591d0ee640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:49:47.554 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:47.571+0000 7f591d0ee640 1 -- 172.21.15.88:0/2511225152 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f5908004c90 con 0x7f59180b0860 2024-03-29T10:49:47.554 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:47.571+0000 7f591d0ee640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:47.555 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:47.571+0000 7f591d0ee640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:49:47.555 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:47.571+0000 7f591d0ee640 10 monclient: dump: 2024-03-29T10:49:47.555 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:49:47.555 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:49:47.555 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:49:47.555 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:49:47.555 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:47.555 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:49:47.555 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:49:47.555 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:49:47.555 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:49:47.555 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:49:47.556 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:47.571+0000 7f591d0ee640 1 -- 172.21.15.88:0/2511225152 <== mon.2 v1:172.21.15.88:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5908004df0 con 0x7f59180b0860 2024-03-29T10:49:47.556 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:47.571+0000 7f591d0ee640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:49:47.556 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:47.571+0000 7f591d0ee640 1 -- 172.21.15.88:0/2511225152 <== mon.2 v1:172.21.15.88:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f5908001410 con 0x7f59180b0860 2024-03-29T10:49:47.556 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:47.571+0000 7f591d0ee640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:47.556 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:47.571+0000 7f591d0ee640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:49:47.556 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:47.571+0000 7f591d0ee640 10 monclient: dump: 2024-03-29T10:49:47.556 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:49:47.556 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:49:47.556 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:49:47.556 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:49:47.556 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:47.556 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:49:47.556 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:49:47.556 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:49:47.557 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:49:47.557 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:49:47.559 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:47.571+0000 7f5920354640 5 monclient: authenticate success, global_id 4586 2024-03-29T10:49:47.559 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:47.571+0000 7f5920354640 10 monclient: get_monmap_and_config success 2024-03-29T10:49:47.559 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:47.571+0000 7f5920354640 4 set_mon_vals no callback set 2024-03-29T10:49:47.559 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:47.572+0000 7f5920354640 10 monclient: shutdown 2024-03-29T10:49:47.559 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:47.572+0000 7f5920354640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:49:47.559 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:47.572+0000 7f5920354640 1 -- 172.21.15.88:0/2511225152 >> v1:172.21.15.88:6790/0 conn(0x7f59180b0860 legacy=0x7f591813e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:47.559 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:47.573+0000 7f5920354640 1 -- 172.21.15.88:0/2511225152 shutdown_connections 2024-03-29T10:49:47.559 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:47.573+0000 7f5920354640 1 -- 172.21.15.88:0/2511225152 >> 172.21.15.88:0/2511225152 conn(0x7f591806b660 msgr2=0x7f591806ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:49:47.559 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:47.573+0000 7f5920354640 1 -- 172.21.15.88:0/2511225152 shutdown_connections 2024-03-29T10:49:47.559 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:47.573+0000 7f5920354640 1 -- 172.21.15.88:0/2511225152 wait complete. 2024-03-29T10:49:47.559 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:47.574+0000 7f5920354640 10 monclient: build_initial_monmap 2024-03-29T10:49:47.559 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:47.574+0000 7f5920354640 10 monclient: monmap: 2024-03-29T10:49:47.559 INFO:teuthology.misc.health.smithi088.stderr:epoch 0 2024-03-29T10:49:47.559 INFO:teuthology.misc.health.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:49:47.560 INFO:teuthology.misc.health.smithi088.stderr:last_changed 0.000000 2024-03-29T10:49:47.560 INFO:teuthology.misc.health.smithi088.stderr:created 0.000000 2024-03-29T10:49:47.560 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 0 (unknown) 2024-03-29T10:49:47.560 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:49:47.560 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-03-29T10:49:47.560 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-29T10:49:47.560 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-c 2024-03-29T10:49:47.560 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:49:47.560 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:47.574+0000 7f5920354640 1 librados: starting msgr at 2024-03-29T10:49:47.560 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:47.574+0000 7f5920354640 1 librados: starting objecter 2024-03-29T10:49:47.560 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:47.574+0000 7f5920354640 1 Processor -- start 2024-03-29T10:49:47.561 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:47.574+0000 7f5920354640 1 -- start start 2024-03-29T10:49:47.561 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:47.574+0000 7f5920354640 1 librados: setting wanted keys 2024-03-29T10:49:47.561 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:47.574+0000 7f5920354640 1 librados: calling monclient init 2024-03-29T10:49:47.561 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:47.574+0000 7f5920354640 10 monclient: init 2024-03-29T10:49:47.561 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:47.575+0000 7f5920354640 10 monclient: _reopen_session rank -1 2024-03-29T10:49:47.561 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:47.575+0000 7f5920354640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:49:47.561 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:47.575+0000 7f5920354640 10 monclient(hunting): picked mon.noname-b con 0x7f59180af8e0 addr v1:172.21.15.196:6789/0 2024-03-29T10:49:47.561 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:47.575+0000 7f5920354640 10 monclient(hunting): picked mon.noname-a con 0x7f59180b0860 addr v1:172.21.15.88:6789/0 2024-03-29T10:49:47.561 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:47.575+0000 7f5920354640 10 monclient(hunting): picked mon.noname-c con 0x7f591813ef30 addr v1:172.21.15.88:6790/0 2024-03-29T10:49:47.561 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:47.575+0000 7f5920354640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f591813b2a0 con 0x7f59180b0860 2024-03-29T10:49:47.561 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:47.575+0000 7f5920354640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f591813b580 con 0x7f59180af8e0 2024-03-29T10:49:47.561 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:47.575+0000 7f5920354640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59181c7330 con 0x7f591813ef30 2024-03-29T10:49:47.561 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:47.575+0000 7f5920354640 10 monclient(hunting): _renew_subs 2024-03-29T10:49:47.561 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:47.575+0000 7f5920354640 10 monclient(hunting): authenticate will time out at 1309.811393s 2024-03-29T10:49:47.562 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:47.580+0000 7f591e0f0640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f59180af8e0 0x7f5918139200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.88:54975/0 (socket says 172.21.15.88:54975) 2024-03-29T10:49:47.562 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:47.580+0000 7f591e0f0640 1 -- 172.21.15.88:0/3896427239 learned_addr learned my addr 172.21.15.88:0/3896427239 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-29T10:49:47.566 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:47.581+0000 7f590effd640 10 client.?.objecter ms_handle_connect 0x7f59180af8e0 2024-03-29T10:49:47.567 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:47.581+0000 7f590effd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:49:47.567 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:47.581+0000 7f590effd640 10 client.?.objecter ms_handle_connect 0x7f591813ef30 2024-03-29T10:49:47.567 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:47.581+0000 7f590effd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:49:47.567 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:47.581+0000 7f590effd640 1 -- 172.21.15.88:0/3896427239 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1299962966 0 0) 0x7f591813b580 con 0x7f59180af8e0 2024-03-29T10:49:47.567 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:47.581+0000 7f590effd640 10 monclient(hunting): _negotiate 2024-03-29T10:49:47.567 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:47.581+0000 7f590effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:47.567 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:47.581+0000 7f590effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:47.567 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:47.581+0000 7f590effd640 10 monclient(hunting): my global_id is 4534 2024-03-29T10:49:47.567 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:47.581+0000 7f590effd640 1 -- 172.21.15.88:0/3896427239 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f58fc004eb0 con 0x7f59180af8e0 2024-03-29T10:49:47.567 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:47.581+0000 7f590effd640 1 -- 172.21.15.88:0/3896427239 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1691604876 0 0) 0x7f59181c7330 con 0x7f591813ef30 2024-03-29T10:49:47.567 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:47.581+0000 7f590effd640 10 monclient(hunting): _negotiate 2024-03-29T10:49:47.567 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:47.581+0000 7f590effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:47.568 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:47.581+0000 7f590effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:47.568 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:47.581+0000 7f590effd640 10 monclient(hunting): my global_id is 4589 2024-03-29T10:49:47.568 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:47.581+0000 7f590effd640 1 -- 172.21.15.88:0/3896427239 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f591813b580 con 0x7f591813ef30 2024-03-29T10:49:47.568 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:47.582+0000 7f590effd640 1 -- 172.21.15.88:0/3896427239 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2322589388 0 0) 0x7f591813b580 con 0x7f591813ef30 2024-03-29T10:49:47.568 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:47.582+0000 7f590effd640 1 -- 172.21.15.88:0/3896427239 --> v1:172.21.15.88:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f59181c7330 con 0x7f591813ef30 2024-03-29T10:49:47.568 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:47.582+0000 7f590effd640 1 -- 172.21.15.88:0/3896427239 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f5910001bb0 con 0x7f591813ef30 2024-03-29T10:49:47.568 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:47.582+0000 7f590effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:47.568 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:47.582+0000 7f590effd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:49:47.568 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:47.582+0000 7f590effd640 10 monclient(hunting): dump: 2024-03-29T10:49:47.568 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:49:47.568 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:49:47.568 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:49:47.568 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:49:47.568 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:47.569 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:49:47.569 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:49:47.569 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:49:47.569 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:49:47.569 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:49:47.569 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:47.582+0000 7f590effd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:49:47.569 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:47.582+0000 7f590effd640 1 -- 172.21.15.88:0/3896427239 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4110871164 0 0) 0x7f58fc004eb0 con 0x7f59180af8e0 2024-03-29T10:49:47.569 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:47.582+0000 7f590effd640 1 -- 172.21.15.88:0/3896427239 --> v1:172.21.15.196:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f591813b580 con 0x7f59180af8e0 2024-03-29T10:49:47.569 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:47.582+0000 7f590effd640 1 -- 172.21.15.88:0/3896427239 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f5908001660 con 0x7f59180af8e0 2024-03-29T10:49:47.569 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:47.582+0000 7f590effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:47.569 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:47.582+0000 7f590effd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T10:49:47.569 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:47.582+0000 7f590effd640 10 monclient(hunting): dump: 2024-03-29T10:49:47.569 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:49:47.569 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:49:47.570 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:49:47.570 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:49:47.570 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:47.570 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:49:47.570 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:49:47.570 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:49:47.570 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:49:47.570 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:49:47.570 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:47.582+0000 7f590effd640 1 -- 172.21.15.88:0/3896427239 <== mon.2 v1:172.21.15.88:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3627463534 0 0) 0x7f59181c7330 con 0x7f591813ef30 2024-03-29T10:49:47.570 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:47.582+0000 7f590effd640 1 -- 172.21.15.88:0/3896427239 >> v1:172.21.15.196:6789/0 conn(0x7f59180af8e0 legacy=0x7f5918139200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:47.570 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:47.582+0000 7f590effd640 1 -- 172.21.15.88:0/3896427239 >> v1:172.21.15.88:6789/0 conn(0x7f59180b0860 legacy=0x7f59181c7850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:47.570 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:47.582+0000 7f590effd640 10 monclient: _finish_hunting 0 2024-03-29T10:49:47.570 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:47.582+0000 7f590effd640 1 monclient: found mon.c 2024-03-29T10:49:47.570 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:47.582+0000 7f590effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:49:47.571 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:47.582+0000 7f590effd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-03-29T10:49:47.571 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:47.582+0000 7f590effd640 1 -- 172.21.15.88:0/3896427239 --> v1:172.21.15.88:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5918139df0 con 0x7f591813ef30 2024-03-29T10:49:47.571 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:47.582+0000 7f590effd640 10 monclient: _finish_auth 0 2024-03-29T10:49:47.571 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:47.582+0000 7f590effd640 10 monclient: _check_auth_tickets 2024-03-29T10:49:47.571 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:47.582+0000 7f590effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:49:47.571 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:47.583+0000 7f5920354640 5 monclient: authenticate success, global_id 4589 2024-03-29T10:49:47.571 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:47.583+0000 7f5920354640 10 monclient: _renew_subs 2024-03-29T10:49:47.571 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:47.583+0000 7f5920354640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-03-29T10:49:47.571 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:47.583+0000 7f5920354640 1 -- 172.21.15.88:0/3896427239 --> v1:172.21.15.88:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f591813a0a0 con 0x7f591813ef30 2024-03-29T10:49:47.571 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:47.583+0000 7f5920354640 10 client.4589.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:49:47.571 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:47.583+0000 7f5920354640 10 monclient: _renew_subs 2024-03-29T10:49:47.571 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:47.583+0000 7f5920354640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-03-29T10:49:47.571 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:47.583+0000 7f5920354640 1 -- 172.21.15.88:0/3896427239 --> v1:172.21.15.88:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f59181e7670 con 0x7f591813ef30 2024-03-29T10:49:47.571 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:47.583+0000 7f5920354640 1 librados: init done 2024-03-29T10:49:47.572 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:47.583+0000 7f590effd640 1 -- 172.21.15.88:0/3896427239 <== mon.2 v1:172.21.15.88:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5910002060 con 0x7f591813ef30 2024-03-29T10:49:47.572 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:47.583+0000 7f590effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:49:47.572 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:47.583+0000 7f590effd640 1 -- 172.21.15.88:0/3896427239 <== mon.2 v1:172.21.15.88:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f5910003b30 con 0x7f591813ef30 2024-03-29T10:49:47.572 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:47.583+0000 7f590effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:47.572 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:47.583+0000 7f590effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:49:47.572 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:47.583+0000 7f590effd640 10 monclient: dump: 2024-03-29T10:49:47.572 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:49:47.572 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:49:47.572 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:49:47.572 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:49:47.572 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:47.572 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:49:47.572 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:49:47.572 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:49:47.572 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:49:47.573 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:49:47.573 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:47.583+0000 7f591c8ed640 4 set_mon_vals no callback set 2024-03-29T10:49:47.573 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:47.584+0000 7f5920354640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:49:47.573 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:47.584+0000 7f5920354640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:49:47.573 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:47.584+0000 7f5920354640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-03-29T10:49:47.573 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:47.584+0000 7f5920354640 1 -- 172.21.15.88:0/3896427239 --> v1:172.21.15.88:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5900005080 con 0x7f591813ef30 2024-03-29T10:49:47.574 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:47.585+0000 7f590effd640 1 -- 172.21.15.88:0/3896427239 <== mon.2 v1:172.21.15.88:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4235629220 0 0) 0x7f5910003c90 con 0x7f591813ef30 2024-03-29T10:49:47.574 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:47.586+0000 7f590effd640 1 -- 172.21.15.88:0/3896427239 <== mon.2 v1:172.21.15.88:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4257673582 0 0) 0x7f5910083aa0 con 0x7f591813ef30 2024-03-29T10:49:47.574 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:47.586+0000 7f590effd640 10 client.4589.objecter ms_dispatch 0x7f5918000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:49:47.574 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:47.586+0000 7f590effd640 3 client.4589.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:49:47.574 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:47.586+0000 7f590effd640 3 client.4589.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:49:47.574 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:47.586+0000 7f590effd640 20 client.4589.objecter dump_active .. 0 homeless 2024-03-29T10:49:47.574 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:47.586+0000 7f591e0f0640 10 monclient: get_auth_request con 0x7f58fc06dec0 auth_method 0 2024-03-29T10:49:47.574 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:47.586+0000 7f590effd640 10 client.4589.objecter ms_handle_connect 0x7f58fc06dec0 2024-03-29T10:49:47.580 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:47.598+0000 7f590effd640 1 -- 172.21.15.88:0/3896427239 <== mon.2 v1:172.21.15.88:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f591004f380 con 0x7f591813ef30 2024-03-29T10:49:47.580 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:47.598+0000 7f590effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:49:47.580 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:47.598+0000 7f590effd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:49:47.897 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:47.914+0000 7f5920354640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:49:47.897 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:47.915+0000 7f5920354640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:49:47.897 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:47.915+0000 7f5920354640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-03-29T10:49:47.897 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:47.915+0000 7f5920354640 1 -- 172.21.15.88:0/3896427239 --> v1:172.21.15.88:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5900005330 con 0x7f591813ef30 2024-03-29T10:49:47.900 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:47.918+0000 7f590effd640 1 -- 172.21.15.88:0/3896427239 <== mon.2 v1:172.21.15.88:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 344506374) 0x7f59100512b0 con 0x7f591813ef30 2024-03-29T10:49:47.900 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:47.918+0000 7f590effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:49:47.900 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:47.918+0000 7f590effd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:49:47.900 INFO:teuthology.misc.health.smithi088.stdout: 2024-03-29T10:49:47.900 INFO:teuthology.misc.health.smithi088.stdout:{"epoch":10,"fsid":"869f5188-f92b-42dd-a46d-8c646e0388a9","created":"2024-03-29T10:40:40.285087+0000","modified":"2024-03-29T10:40:42.536230+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:40:42.536230+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a26c0c1c-8de0-438c-a3f6-3df10ea61574","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5de24915-8dec-49b6-8b16-f3170197c591","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ac054af3-d589-4565-9644-e6af75975c53","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3ec4f045-c714-4a22-bd1a-e07330b45ab3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"cddb9867-8e30-4951-9b13-3d1e78b9c45a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dc6d9911-785d-432f-8b53-56a1ccab302f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"77c88437-b587-496e-8b4d-1824ea8b5305","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ce963d96-5a8e-44c3-99de-c0dbebe234ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:49:47.904 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:47.922+0000 7f5920354640 10 librados: watch_flush enter 2024-03-29T10:49:47.904 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:47.922+0000 7f5920354640 10 librados: watch_flush exit 2024-03-29T10:49:47.904 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:47.922+0000 7f5920354640 20 client.4589.objecter shutdown clearing up homeless session... 2024-03-29T10:49:47.904 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:47.922+0000 7f5920354640 10 client.4589.objecter successfully canceled tick 2024-03-29T10:49:47.904 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:47.922+0000 7f5920354640 1 -- 172.21.15.88:0/3896427239 >> v1:172.21.15.88:6816/1752861349 conn(0x7f58fc06dec0 legacy=0x7f58fc070380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:47.905 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:47.922+0000 7f5920354640 10 monclient: shutdown 2024-03-29T10:49:47.905 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:47.922+0000 7f5920354640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:49:47.905 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:47.922+0000 7f5920354640 1 -- 172.21.15.88:0/3896427239 >> v1:172.21.15.88:6790/0 conn(0x7f591813ef30 legacy=0x7f59181c7f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:47.905 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:47.922+0000 7f5920354640 1 -- 172.21.15.88:0/3896427239 shutdown_connections 2024-03-29T10:49:47.905 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:47.922+0000 7f5920354640 1 -- 172.21.15.88:0/3896427239 >> 172.21.15.88:0/3896427239 conn(0x7f591806b660 msgr2=0x7f591806ba40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:49:47.905 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:47.923+0000 7f5920354640 1 -- 172.21.15.88:0/3896427239 shutdown_connections 2024-03-29T10:49:47.905 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:47.923+0000 7f5920354640 1 -- 172.21.15.88:0/3896427239 wait complete. 2024-03-29T10:49:47.905 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:47.924+0000 7f5920354640 1 librados: shutdown 2024-03-29T10:49:47.922 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:49:54.728 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:49:54.878 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:54.895+0000 7f1310379640 10 monclient: get_monmap_and_config 2024-03-29T10:49:54.878 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:54.895+0000 7f1310379640 10 monclient: build_initial_monmap 2024-03-29T10:49:54.878 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:54.895+0000 7f1310379640 10 monclient: monmap: 2024-03-29T10:49:54.878 INFO:teuthology.misc.health.smithi088.stderr:epoch 0 2024-03-29T10:49:54.878 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:49:54.878 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:49:54.896638+0000 2024-03-29T10:49:54.878 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:49:54.896638+0000 2024-03-29T10:49:54.879 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 0 (unknown) 2024-03-29T10:49:54.879 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:49:54.879 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-03-29T10:49:54.879 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-29T10:49:54.879 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-b 2024-03-29T10:49:54.879 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:49:54.880 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:54.898+0000 7f1310379640 1 Processor -- start 2024-03-29T10:49:54.880 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:54.898+0000 7f1310379640 1 -- start start 2024-03-29T10:49:54.880 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:54.898+0000 7f1310379640 10 monclient: init 2024-03-29T10:49:54.882 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:54.899+0000 7f1310379640 10 monclient: _reopen_session rank -1 2024-03-29T10:49:54.882 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:54.899+0000 7f1310379640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:49:54.882 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:54.899+0000 7f1310379640 10 monclient(hunting): picked mon.noname-a con 0x7f13080b4100 addr v1:172.21.15.88:6789/0 2024-03-29T10:49:54.882 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:54.899+0000 7f1310379640 10 monclient(hunting): picked mon.noname-c con 0x7f130813ee90 addr v1:172.21.15.196:6789/0 2024-03-29T10:49:54.882 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:54.899+0000 7f1310379640 10 monclient(hunting): picked mon.noname-b con 0x7f13080af900 addr v1:172.21.15.88:6790/0 2024-03-29T10:49:54.882 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:54.899+0000 7f1310379640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13080afd10 con 0x7f13080b4100 2024-03-29T10:49:54.882 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:54.899+0000 7f1310379640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13080af6f0 con 0x7f130813ee90 2024-03-29T10:49:54.882 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:54.899+0000 7f1310379640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13080aaf60 con 0x7f13080af900 2024-03-29T10:49:54.882 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:54.899+0000 7f1310379640 10 monclient(hunting): _renew_subs 2024-03-29T10:49:54.882 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:54.899+0000 7f1310379640 10 monclient(hunting): authenticate will time out at 1317.135854s 2024-03-29T10:49:54.886 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:54.904+0000 7f130d914640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f130813ee90 0x7f13080adfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.88:56961/0 (socket says 172.21.15.88:56961) 2024-03-29T10:49:54.886 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:54.904+0000 7f130d914640 1 -- 172.21.15.88:0/644027444 learned_addr learned my addr 172.21.15.88:0/644027444 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-29T10:49:54.887 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:54.905+0000 7f130d113640 1 -- 172.21.15.88:0/644027444 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 124407728 0 0) 0x7f13080af6f0 con 0x7f130813ee90 2024-03-29T10:49:54.887 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:54.905+0000 7f130d113640 10 monclient(hunting): _negotiate 2024-03-29T10:49:54.887 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:54.905+0000 7f130d113640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:54.887 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:54.905+0000 7f130d113640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:54.887 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:54.905+0000 7f130d113640 10 monclient(hunting): my global_id is 4537 2024-03-29T10:49:54.887 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:54.905+0000 7f130d113640 1 -- 172.21.15.88:0/644027444 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f12f4003520 con 0x7f130813ee90 2024-03-29T10:49:54.888 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:54.906+0000 7f130d113640 1 -- 172.21.15.88:0/644027444 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3243321265 0 0) 0x7f12f4003520 con 0x7f130813ee90 2024-03-29T10:49:54.888 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:54.906+0000 7f130d113640 1 -- 172.21.15.88:0/644027444 >> v1:172.21.15.88:6790/0 conn(0x7f13080af900 legacy=0x7f130813b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:54.888 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:54.906+0000 7f130d113640 1 -- 172.21.15.88:0/644027444 >> v1:172.21.15.88:6789/0 conn(0x7f13080b4100 legacy=0x7f13080a9850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:49:54.888 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:54.906+0000 7f130d113640 10 monclient: _finish_hunting 0 2024-03-29T10:49:54.888 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:54.906+0000 7f130d113640 1 monclient: found mon.noname-c 2024-03-29T10:49:54.888 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:54.906+0000 7f130d113640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:49:54.888 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:54.906+0000 7f130d113640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.196:6789/0 2024-03-29T10:49:54.888 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:54.906+0000 7f130d113640 1 -- 172.21.15.88:0/644027444 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f130813e970 con 0x7f130813ee90 2024-03-29T10:49:54.888 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:54.906+0000 7f130d113640 10 monclient: _finish_auth 0 2024-03-29T10:49:54.888 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:54.906+0000 7f130d113640 10 monclient: _check_auth_tickets 2024-03-29T10:49:54.889 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:54.906+0000 7f130d113640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:49:54.889 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:54.906+0000 7f130d113640 1 -- 172.21.15.88:0/644027444 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f12f8001690 con 0x7f130813ee90 2024-03-29T10:49:54.889 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:54.906+0000 7f1310379640 5 monclient: authenticate success, global_id 4537 2024-03-29T10:49:54.889 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:54.906+0000 7f1310379640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:49:54.889 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:54.906+0000 7f130d113640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:54.894 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:54.906+0000 7f130d113640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:49:54.894 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:54.906+0000 7f130d113640 10 monclient: dump: 2024-03-29T10:49:54.894 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:49:54.894 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:49:54.894 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:49:54.894 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:49:54.895 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:54.895 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:49:54.895 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:49:54.895 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:49:54.895 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:49:54.895 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:49:54.895 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:54.906+0000 7f130d113640 1 -- 172.21.15.88:0/644027444 <== mon.1 v1:172.21.15.196:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f12f8001920 con 0x7f130813ee90 2024-03-29T10:49:54.895 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:54.906+0000 7f130d113640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:49:54.895 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:54.906+0000 7f130d113640 1 -- 172.21.15.88:0/644027444 <== mon.1 v1:172.21.15.196:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f12f8001cb0 con 0x7f130813ee90 2024-03-29T10:49:54.895 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:54.906+0000 7f130d113640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:54.895 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:54.906+0000 7f130d113640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:49:54.895 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:54.906+0000 7f130d113640 10 monclient: dump: 2024-03-29T10:49:54.895 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:49:54.896 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:49:54.896 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:49:54.896 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:49:54.896 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:54.896 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:49:54.896 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:49:54.896 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:49:54.896 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:49:54.896 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:49:54.896 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:54.906+0000 7f1310379640 10 monclient: get_monmap_and_config success 2024-03-29T10:49:54.896 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:54.906+0000 7f1310379640 4 set_mon_vals no callback set 2024-03-29T10:49:54.897 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:54.907+0000 7f1310379640 10 monclient: shutdown 2024-03-29T10:49:54.897 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:54.907+0000 7f1310379640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:49:54.897 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:54.907+0000 7f1310379640 1 -- 172.21.15.88:0/644027444 >> v1:172.21.15.196:6789/0 conn(0x7f130813ee90 legacy=0x7f13080adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:54.897 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:54.909+0000 7f1310379640 1 -- 172.21.15.88:0/644027444 shutdown_connections 2024-03-29T10:49:54.897 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:54.909+0000 7f1310379640 1 -- 172.21.15.88:0/644027444 >> 172.21.15.88:0/644027444 conn(0x7f13080b35e0 msgr2=0x7f13080b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:49:54.897 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:54.910+0000 7f1310379640 1 -- 172.21.15.88:0/644027444 shutdown_connections 2024-03-29T10:49:54.897 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:54.911+0000 7f1310379640 1 -- 172.21.15.88:0/644027444 wait complete. 2024-03-29T10:49:54.900 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:54.913+0000 7f1310379640 10 monclient: build_initial_monmap 2024-03-29T10:49:54.900 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:54.916+0000 7f1310379640 10 monclient: monmap: 2024-03-29T10:49:54.900 INFO:teuthology.misc.health.smithi088.stderr:epoch 0 2024-03-29T10:49:54.900 INFO:teuthology.misc.health.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:49:54.900 INFO:teuthology.misc.health.smithi088.stderr:last_changed 0.000000 2024-03-29T10:49:54.900 INFO:teuthology.misc.health.smithi088.stderr:created 0.000000 2024-03-29T10:49:54.901 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 0 (unknown) 2024-03-29T10:49:54.901 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:49:54.901 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-03-29T10:49:54.901 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-29T10:49:54.901 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-c 2024-03-29T10:49:54.901 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:49:54.901 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:54.916+0000 7f1310379640 1 librados: starting msgr at 2024-03-29T10:49:54.901 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:54.916+0000 7f1310379640 1 librados: starting objecter 2024-03-29T10:49:54.901 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:54.916+0000 7f1310379640 1 Processor -- start 2024-03-29T10:49:54.901 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:54.917+0000 7f1310379640 1 -- start start 2024-03-29T10:49:54.901 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:54.917+0000 7f1310379640 1 librados: setting wanted keys 2024-03-29T10:49:54.901 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:54.917+0000 7f1310379640 1 librados: calling monclient init 2024-03-29T10:49:54.901 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:54.917+0000 7f1310379640 10 monclient: init 2024-03-29T10:49:54.901 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:54.917+0000 7f1310379640 10 monclient: _reopen_session rank -1 2024-03-29T10:49:54.902 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:54.917+0000 7f1310379640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:49:54.902 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:54.917+0000 7f1310379640 10 monclient(hunting): picked mon.noname-b con 0x7f13080af900 addr v1:172.21.15.196:6789/0 2024-03-29T10:49:54.902 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:54.917+0000 7f1310379640 10 monclient(hunting): picked mon.noname-a con 0x7f13080b4100 addr v1:172.21.15.88:6789/0 2024-03-29T10:49:54.902 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:54.917+0000 7f1310379640 10 monclient(hunting): picked mon.noname-c con 0x7f13080723d0 addr v1:172.21.15.88:6790/0 2024-03-29T10:49:54.902 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:54.917+0000 7f1310379640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1308070380 con 0x7f13080b4100 2024-03-29T10:49:54.902 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:54.918+0000 7f1310379640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1308070660 con 0x7f13080af900 2024-03-29T10:49:54.902 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:54.918+0000 7f1310379640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13081420c0 con 0x7f13080723d0 2024-03-29T10:49:54.902 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:54.918+0000 7f1310379640 10 monclient(hunting): _renew_subs 2024-03-29T10:49:54.902 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:54.918+0000 7f1310379640 10 monclient(hunting): authenticate will time out at 1317.153910s 2024-03-29T10:49:54.904 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:54.922+0000 7f130d914640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f13080b4100 0x7f1308070a80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:33549/0 (socket says 172.21.15.88:33549) 2024-03-29T10:49:54.904 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:54.922+0000 7f130d914640 1 -- 172.21.15.88:0/3929526031 learned_addr learned my addr 172.21.15.88:0/3929526031 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-29T10:49:54.904 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:54.922+0000 7f12feffd640 10 client.?.objecter ms_handle_connect 0x7f13080b4100 2024-03-29T10:49:54.904 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:54.922+0000 7f12feffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:49:54.904 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:54.922+0000 7f12feffd640 1 -- 172.21.15.88:0/3929526031 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1785077364 0 0) 0x7f1308070380 con 0x7f13080b4100 2024-03-29T10:49:54.905 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:54.922+0000 7f12feffd640 10 monclient(hunting): _negotiate 2024-03-29T10:49:54.905 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:54.922+0000 7f12feffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:54.905 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:54.922+0000 7f12feffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:54.905 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:54.922+0000 7f12feffd640 10 monclient(hunting): my global_id is 4563 2024-03-29T10:49:54.905 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:54.922+0000 7f12feffd640 1 -- 172.21.15.88:0/3929526031 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f12ec004eb0 con 0x7f13080b4100 2024-03-29T10:49:54.907 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:54.924+0000 7f12feffd640 1 -- 172.21.15.88:0/3929526031 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3104309592 0 0) 0x7f12ec004eb0 con 0x7f13080b4100 2024-03-29T10:49:54.908 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:54.924+0000 7f12feffd640 1 -- 172.21.15.88:0/3929526031 --> v1:172.21.15.88:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1308070380 con 0x7f13080b4100 2024-03-29T10:49:54.908 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:54.924+0000 7f12feffd640 1 -- 172.21.15.88:0/3929526031 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f12f8001550 con 0x7f13080b4100 2024-03-29T10:49:54.908 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:54.925+0000 7f12feffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:54.908 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:54.925+0000 7f12feffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:49:54.908 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:54.925+0000 7f12feffd640 10 monclient(hunting): dump: 2024-03-29T10:49:54.908 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:49:54.908 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:49:54.908 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:49:54.908 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:49:54.908 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:54.908 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:49:54.908 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:49:54.908 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:49:54.908 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:49:54.909 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:49:54.909 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:54.925+0000 7f12feffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:49:54.909 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:54.925+0000 7f12feffd640 10 client.?.objecter ms_handle_connect 0x7f13080af900 2024-03-29T10:49:54.909 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:54.925+0000 7f12feffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:49:54.909 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:54.925+0000 7f12feffd640 10 client.?.objecter ms_handle_connect 0x7f13080723d0 2024-03-29T10:49:54.909 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:54.925+0000 7f12feffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:49:54.909 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:54.925+0000 7f12feffd640 1 -- 172.21.15.88:0/3929526031 <== mon.0 v1:172.21.15.88:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 687517739 0 0) 0x7f1308070380 con 0x7f13080b4100 2024-03-29T10:49:54.909 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:54.925+0000 7f12feffd640 1 -- 172.21.15.88:0/3929526031 >> v1:172.21.15.88:6790/0 conn(0x7f13080723d0 legacy=0x7f1308071190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:54.909 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:54.925+0000 7f12feffd640 1 -- 172.21.15.88:0/3929526031 >> v1:172.21.15.196:6789/0 conn(0x7f13080af900 legacy=0x7f13081df2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:54.909 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:54.925+0000 7f12feffd640 10 monclient: _finish_hunting 0 2024-03-29T10:49:54.909 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:54.925+0000 7f12feffd640 1 monclient: found mon.a 2024-03-29T10:49:54.910 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:54.927+0000 7f12feffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:49:54.910 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:54.927+0000 7f12feffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-03-29T10:49:54.910 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:54.927+0000 7f12feffd640 1 -- 172.21.15.88:0/3929526031 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13080718a0 con 0x7f13080b4100 2024-03-29T10:49:54.910 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:54.927+0000 7f12feffd640 10 monclient: _finish_auth 0 2024-03-29T10:49:54.910 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:54.927+0000 7f12feffd640 10 monclient: _check_auth_tickets 2024-03-29T10:49:54.910 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:54.927+0000 7f12feffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:49:54.910 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:54.927+0000 7f1310379640 5 monclient: authenticate success, global_id 4563 2024-03-29T10:49:54.910 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:54.927+0000 7f1310379640 10 monclient: _renew_subs 2024-03-29T10:49:54.910 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:54.927+0000 7f1310379640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-03-29T10:49:54.910 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:54.927+0000 7f1310379640 1 -- 172.21.15.88:0/3929526031 --> v1:172.21.15.88:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13081df9c0 con 0x7f13080b4100 2024-03-29T10:49:54.910 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:54.927+0000 7f1310379640 10 client.4563.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:49:54.911 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:54.927+0000 7f1310379640 10 monclient: _renew_subs 2024-03-29T10:49:54.911 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:54.927+0000 7f1310379640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-03-29T10:49:54.911 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:54.927+0000 7f1310379640 1 -- 172.21.15.88:0/3929526031 --> v1:172.21.15.88:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1308071c90 con 0x7f13080b4100 2024-03-29T10:49:54.911 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:54.927+0000 7f1310379640 1 librados: init done 2024-03-29T10:49:54.911 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:54.929+0000 7f1310379640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:49:54.911 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:54.929+0000 7f1310379640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:49:54.911 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:54.929+0000 7f1310379640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-03-29T10:49:54.911 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:54.929+0000 7f1310379640 1 -- 172.21.15.88:0/3929526031 --> v1:172.21.15.88:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1308142250 con 0x7f13080b4100 2024-03-29T10:49:54.915 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:54.931+0000 7f12feffd640 1 -- 172.21.15.88:0/3929526031 <== mon.0 v1:172.21.15.88:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f12f8004bf0 con 0x7f13080b4100 2024-03-29T10:49:54.915 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:54.931+0000 7f12feffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:49:54.915 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:54.931+0000 7f12feffd640 1 -- 172.21.15.88:0/3929526031 <== mon.0 v1:172.21.15.88:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f12f8004d60 con 0x7f13080b4100 2024-03-29T10:49:54.915 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:54.931+0000 7f12feffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:54.915 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:54.931+0000 7f12feffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:49:54.915 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:54.931+0000 7f12feffd640 10 monclient: dump: 2024-03-29T10:49:54.915 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:49:54.915 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:49:54.915 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:49:54.915 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:49:54.915 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:54.915 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:49:54.915 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:49:54.915 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:49:54.916 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:49:54.916 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:49:54.916 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:54.931+0000 7f12fffff640 4 set_mon_vals no callback set 2024-03-29T10:49:54.916 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:54.933+0000 7f12feffd640 1 -- 172.21.15.88:0/3929526031 <== mon.0 v1:172.21.15.88:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4235629220 0 0) 0x7f12f801ad30 con 0x7f13080b4100 2024-03-29T10:49:54.916 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:54.934+0000 7f12feffd640 1 -- 172.21.15.88:0/3929526031 <== mon.0 v1:172.21.15.88:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4257673582 0 0) 0x7f12f8054890 con 0x7f13080b4100 2024-03-29T10:49:54.916 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:54.934+0000 7f12feffd640 10 client.4563.objecter ms_dispatch 0x7f1308000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:49:54.916 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:54.934+0000 7f12feffd640 3 client.4563.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:49:54.917 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:54.934+0000 7f12feffd640 3 client.4563.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:49:54.917 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:54.934+0000 7f12feffd640 20 client.4563.objecter dump_active .. 0 homeless 2024-03-29T10:49:54.917 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:54.934+0000 7f130e115640 10 monclient: get_auth_request con 0x7f12ec06db30 auth_method 0 2024-03-29T10:49:54.917 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:54.935+0000 7f12feffd640 10 client.4563.objecter ms_handle_connect 0x7f12ec06db30 2024-03-29T10:49:54.923 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:54.941+0000 7f12feffd640 1 -- 172.21.15.88:0/3929526031 <== mon.0 v1:172.21.15.88:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f12f8050b30 con 0x7f13080b4100 2024-03-29T10:49:54.923 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:54.941+0000 7f12feffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:49:54.923 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:54.941+0000 7f12feffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:49:55.250 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:55.265+0000 7f1310379640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:49:55.251 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:55.265+0000 7f1310379640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:49:55.251 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:55.265+0000 7f1310379640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-03-29T10:49:55.251 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:55.265+0000 7f1310379640 1 -- 172.21.15.88:0/3929526031 --> v1:172.21.15.88:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f130813f300 con 0x7f13080b4100 2024-03-29T10:49:55.252 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:55.270+0000 7f12feffd640 1 -- 172.21.15.88:0/3929526031 <== mon.0 v1:172.21.15.88:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 344506374) 0x7f12f8050b30 con 0x7f13080b4100 2024-03-29T10:49:55.252 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:55.270+0000 7f12feffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:49:55.252 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:55.270+0000 7f12feffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:49:55.255 INFO:teuthology.misc.health.smithi088.stdout: 2024-03-29T10:49:55.255 INFO:teuthology.misc.health.smithi088.stdout:{"epoch":10,"fsid":"869f5188-f92b-42dd-a46d-8c646e0388a9","created":"2024-03-29T10:40:40.285087+0000","modified":"2024-03-29T10:40:42.536230+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:40:42.536230+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a26c0c1c-8de0-438c-a3f6-3df10ea61574","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5de24915-8dec-49b6-8b16-f3170197c591","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ac054af3-d589-4565-9644-e6af75975c53","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3ec4f045-c714-4a22-bd1a-e07330b45ab3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"cddb9867-8e30-4951-9b13-3d1e78b9c45a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dc6d9911-785d-432f-8b53-56a1ccab302f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"77c88437-b587-496e-8b4d-1824ea8b5305","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ce963d96-5a8e-44c3-99de-c0dbebe234ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:49:55.255 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:55.273+0000 7f1310379640 10 librados: watch_flush enter 2024-03-29T10:49:55.256 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:55.273+0000 7f1310379640 10 librados: watch_flush exit 2024-03-29T10:49:55.256 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:55.274+0000 7f1310379640 20 client.4563.objecter shutdown clearing up homeless session... 2024-03-29T10:49:55.256 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:55.274+0000 7f1310379640 10 client.4563.objecter successfully canceled tick 2024-03-29T10:49:55.256 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:55.274+0000 7f1310379640 1 -- 172.21.15.88:0/3929526031 >> v1:172.21.15.88:6816/1752861349 conn(0x7f12ec06db30 legacy=0x7f12ec06fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:55.256 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:55.274+0000 7f1310379640 10 monclient: shutdown 2024-03-29T10:49:55.256 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:55.274+0000 7f1310379640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:49:55.256 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:55.274+0000 7f1310379640 1 -- 172.21.15.88:0/3929526031 >> v1:172.21.15.88:6789/0 conn(0x7f13080b4100 legacy=0x7f1308070a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:55.257 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:55.274+0000 7f1310379640 1 -- 172.21.15.88:0/3929526031 shutdown_connections 2024-03-29T10:49:55.257 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:55.274+0000 7f1310379640 1 -- 172.21.15.88:0/3929526031 >> 172.21.15.88:0/3929526031 conn(0x7f13080b35e0 msgr2=0x7f13080b2640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:49:55.257 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:55.275+0000 7f1310379640 1 -- 172.21.15.88:0/3929526031 shutdown_connections 2024-03-29T10:49:55.257 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:55.275+0000 7f1310379640 1 -- 172.21.15.88:0/3929526031 wait complete. 2024-03-29T10:49:55.257 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:49:55.275+0000 7f1310379640 1 librados: shutdown 2024-03-29T10:49:55.276 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:50:02.083 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:50:02.245 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:02.263+0000 7f7ddcf7b640 10 monclient: get_monmap_and_config 2024-03-29T10:50:02.245 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:02.263+0000 7f7ddcf7b640 10 monclient: build_initial_monmap 2024-03-29T10:50:02.245 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:02.263+0000 7f7ddcf7b640 10 monclient: monmap: 2024-03-29T10:50:02.246 INFO:teuthology.misc.health.smithi088.stderr:epoch 0 2024-03-29T10:50:02.246 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:50:02.246 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:50:02.264484+0000 2024-03-29T10:50:02.246 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:50:02.264484+0000 2024-03-29T10:50:02.246 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 0 (unknown) 2024-03-29T10:50:02.246 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:50:02.246 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-03-29T10:50:02.246 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-29T10:50:02.246 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-b 2024-03-29T10:50:02.246 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:50:02.248 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:02.266+0000 7f7ddcf7b640 1 Processor -- start 2024-03-29T10:50:02.248 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:02.266+0000 7f7ddcf7b640 1 -- start start 2024-03-29T10:50:02.248 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:02.266+0000 7f7ddcf7b640 10 monclient: init 2024-03-29T10:50:02.252 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:02.267+0000 7f7ddcf7b640 10 monclient: _reopen_session rank -1 2024-03-29T10:50:02.252 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:02.267+0000 7f7ddcf7b640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:50:02.252 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:02.267+0000 7f7ddcf7b640 10 monclient(hunting): picked mon.noname-a con 0x7f7dd80a9750 addr v1:172.21.15.88:6789/0 2024-03-29T10:50:02.253 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:02.271+0000 7f7ddcf7b640 10 monclient(hunting): picked mon.noname-c con 0x7f7dd813e1e0 addr v1:172.21.15.196:6789/0 2024-03-29T10:50:02.253 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:02.271+0000 7f7ddcf7b640 10 monclient(hunting): picked mon.noname-b con 0x7f7dd8071c30 addr v1:172.21.15.88:6790/0 2024-03-29T10:50:02.254 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:02.271+0000 7f7dd5d9b640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f7dd813e1e0 0x7f7dd80ac710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.88:47577/0 (socket says 172.21.15.88:47577) 2024-03-29T10:50:02.254 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:02.271+0000 7f7dd5d9b640 1 -- 172.21.15.88:0/1501158067 learned_addr learned my addr 172.21.15.88:0/1501158067 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-29T10:50:02.254 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:02.271+0000 7f7ddcf7b640 1 -- 172.21.15.88:0/1501158067 --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7dd80ab300 con 0x7f7dd80a9750 2024-03-29T10:50:02.254 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:02.271+0000 7f7ddcf7b640 1 -- 172.21.15.88:0/1501158067 --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7dd80ac2f0 con 0x7f7dd813e1e0 2024-03-29T10:50:02.254 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:02.271+0000 7f7ddcf7b640 1 -- 172.21.15.88:0/1501158067 --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7dd80ad8d0 con 0x7f7dd8071c30 2024-03-29T10:50:02.254 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:02.271+0000 7f7ddcf7b640 10 monclient(hunting): _renew_subs 2024-03-29T10:50:02.254 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:02.272+0000 7f7ddcf7b640 10 monclient(hunting): authenticate will time out at 1324.507947s 2024-03-29T10:50:02.258 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:02.275+0000 7f7dd559a640 1 -- 172.21.15.88:0/1501158067 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3532617319 0 0) 0x7f7dd80ac2f0 con 0x7f7dd813e1e0 2024-03-29T10:50:02.258 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:02.275+0000 7f7dd559a640 10 monclient(hunting): _negotiate 2024-03-29T10:50:02.258 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:02.275+0000 7f7dd559a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:50:02.258 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:02.275+0000 7f7dd559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:50:02.258 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:02.275+0000 7f7dd559a640 10 monclient(hunting): my global_id is 4543 2024-03-29T10:50:02.258 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:02.275+0000 7f7dd559a640 1 -- 172.21.15.88:0/1501158067 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7dc0004630 con 0x7f7dd813e1e0 2024-03-29T10:50:02.258 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:02.275+0000 7f7dd559a640 1 -- 172.21.15.88:0/1501158067 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3286959173 0 0) 0x7f7dc0004630 con 0x7f7dd813e1e0 2024-03-29T10:50:02.258 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:02.275+0000 7f7dd559a640 1 -- 172.21.15.88:0/1501158067 >> v1:172.21.15.88:6790/0 conn(0x7f7dd8071c30 legacy=0x7f7dd80ace20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:02.259 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:02.275+0000 7f7dd559a640 1 -- 172.21.15.88:0/1501158067 >> v1:172.21.15.88:6789/0 conn(0x7f7dd80a9750 legacy=0x7f7dd813dad0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:50:02.259 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:02.275+0000 7f7dd559a640 10 monclient: _finish_hunting 0 2024-03-29T10:50:02.259 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:02.275+0000 7f7dd559a640 1 monclient: found mon.noname-c 2024-03-29T10:50:02.259 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:02.276+0000 7f7dd559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:50:02.259 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:02.276+0000 7f7dd559a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.196:6789/0 2024-03-29T10:50:02.259 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:02.276+0000 7f7dd559a640 1 -- 172.21.15.88:0/1501158067 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7dd80ada60 con 0x7f7dd813e1e0 2024-03-29T10:50:02.259 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:02.276+0000 7f7dd559a640 10 monclient: _finish_auth 0 2024-03-29T10:50:02.259 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:02.276+0000 7f7dd559a640 10 monclient: _check_auth_tickets 2024-03-29T10:50:02.259 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:02.276+0000 7f7dd559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:50:02.259 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:02.276+0000 7f7dd559a640 1 -- 172.21.15.88:0/1501158067 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f7dcc001690 con 0x7f7dd813e1e0 2024-03-29T10:50:02.259 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:02.276+0000 7f7dd559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:50:02.259 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:02.276+0000 7f7dd559a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:50:02.260 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:02.276+0000 7f7dd559a640 10 monclient: dump: 2024-03-29T10:50:02.260 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:50:02.260 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:50:02.260 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:50:02.260 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:50:02.260 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:50:02.260 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:50:02.260 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:50:02.260 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:50:02.261 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:50:02.261 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:50:02.261 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:02.276+0000 7f7ddcf7b640 5 monclient: authenticate success, global_id 4543 2024-03-29T10:50:02.261 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:02.276+0000 7f7ddcf7b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:50:02.262 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:02.277+0000 7f7dd559a640 1 -- 172.21.15.88:0/1501158067 <== mon.1 v1:172.21.15.196:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7dcc001920 con 0x7f7dd813e1e0 2024-03-29T10:50:02.262 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:02.277+0000 7f7dd559a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:50:02.262 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:02.277+0000 7f7dd559a640 1 -- 172.21.15.88:0/1501158067 <== mon.1 v1:172.21.15.196:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f7dcc001cb0 con 0x7f7dd813e1e0 2024-03-29T10:50:02.262 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:02.277+0000 7f7dd559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:50:02.262 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:02.277+0000 7f7dd559a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:50:02.262 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:02.277+0000 7f7dd559a640 10 monclient: dump: 2024-03-29T10:50:02.262 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:50:02.262 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:50:02.263 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:50:02.263 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:50:02.263 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:50:02.263 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:50:02.263 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:50:02.263 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:50:02.263 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:50:02.263 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:50:02.263 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:02.277+0000 7f7ddcf7b640 10 monclient: get_monmap_and_config success 2024-03-29T10:50:02.263 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:02.277+0000 7f7ddcf7b640 4 set_mon_vals no callback set 2024-03-29T10:50:02.263 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:02.277+0000 7f7ddcf7b640 10 monclient: shutdown 2024-03-29T10:50:02.263 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:02.277+0000 7f7ddcf7b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:50:02.263 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:02.277+0000 7f7ddcf7b640 1 -- 172.21.15.88:0/1501158067 >> v1:172.21.15.196:6789/0 conn(0x7f7dd813e1e0 legacy=0x7f7dd80ac710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:02.264 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:02.279+0000 7f7ddcf7b640 1 -- 172.21.15.88:0/1501158067 shutdown_connections 2024-03-29T10:50:02.264 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:02.279+0000 7f7ddcf7b640 1 -- 172.21.15.88:0/1501158067 >> 172.21.15.88:0/1501158067 conn(0x7f7dd806b660 msgr2=0x7f7dd806ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:50:02.264 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:02.279+0000 7f7ddcf7b640 1 -- 172.21.15.88:0/1501158067 shutdown_connections 2024-03-29T10:50:02.264 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:02.279+0000 7f7ddcf7b640 1 -- 172.21.15.88:0/1501158067 wait complete. 2024-03-29T10:50:02.265 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:02.280+0000 7f7ddcf7b640 10 monclient: build_initial_monmap 2024-03-29T10:50:02.265 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:02.281+0000 7f7ddcf7b640 10 monclient: monmap: 2024-03-29T10:50:02.265 INFO:teuthology.misc.health.smithi088.stderr:epoch 0 2024-03-29T10:50:02.265 INFO:teuthology.misc.health.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:50:02.265 INFO:teuthology.misc.health.smithi088.stderr:last_changed 0.000000 2024-03-29T10:50:02.265 INFO:teuthology.misc.health.smithi088.stderr:created 0.000000 2024-03-29T10:50:02.265 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 0 (unknown) 2024-03-29T10:50:02.266 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:50:02.266 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-03-29T10:50:02.266 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-29T10:50:02.266 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-c 2024-03-29T10:50:02.266 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:50:02.266 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:02.281+0000 7f7ddcf7b640 1 librados: starting msgr at 2024-03-29T10:50:02.266 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:02.281+0000 7f7ddcf7b640 1 librados: starting objecter 2024-03-29T10:50:02.266 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:02.282+0000 7f7ddcf7b640 1 Processor -- start 2024-03-29T10:50:02.266 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:02.282+0000 7f7ddcf7b640 1 -- start start 2024-03-29T10:50:02.266 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:02.282+0000 7f7ddcf7b640 1 librados: setting wanted keys 2024-03-29T10:50:02.266 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:02.282+0000 7f7ddcf7b640 1 librados: calling monclient init 2024-03-29T10:50:02.266 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:02.282+0000 7f7ddcf7b640 10 monclient: init 2024-03-29T10:50:02.267 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:02.283+0000 7f7ddcf7b640 10 monclient: _reopen_session rank -1 2024-03-29T10:50:02.267 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:02.283+0000 7f7ddcf7b640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:50:02.267 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:02.283+0000 7f7ddcf7b640 10 monclient(hunting): picked mon.noname-c con 0x7f7dd8071c30 addr v1:172.21.15.88:6790/0 2024-03-29T10:50:02.267 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:02.285+0000 7f7ddcf7b640 10 monclient(hunting): picked mon.noname-a con 0x7f7dd80a9750 addr v1:172.21.15.88:6789/0 2024-03-29T10:50:02.267 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:02.285+0000 7f7ddcf7b640 10 monclient(hunting): picked mon.noname-b con 0x7f7dd813e1e0 addr v1:172.21.15.196:6789/0 2024-03-29T10:50:02.267 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:02.285+0000 7f7ddcf7b640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7dd80b2c40 con 0x7f7dd80a9750 2024-03-29T10:50:02.267 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:02.285+0000 7f7ddcf7b640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7dd80b3270 con 0x7f7dd813e1e0 2024-03-29T10:50:02.267 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:02.285+0000 7f7ddcf7b640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7dd80b2360 con 0x7f7dd8071c30 2024-03-29T10:50:02.267 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:02.285+0000 7f7ddcf7b640 10 monclient(hunting): _renew_subs 2024-03-29T10:50:02.268 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:02.285+0000 7f7ddcf7b640 10 monclient(hunting): authenticate will time out at 1324.521136s 2024-03-29T10:50:02.270 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:02.288+0000 7f7dd5d9b640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f7dd80a9750 0x7f7dd81307d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:57917/0 (socket says 172.21.15.88:57917) 2024-03-29T10:50:02.270 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:02.288+0000 7f7dd5d9b640 1 -- 172.21.15.88:0/1223098751 learned_addr learned my addr 172.21.15.88:0/1223098751 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-29T10:50:02.270 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:02.288+0000 7f7dc6ffd640 10 client.?.objecter ms_handle_connect 0x7f7dd80a9750 2024-03-29T10:50:02.270 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:02.288+0000 7f7dc6ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:50:02.272 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:02.289+0000 7f7dc6ffd640 1 -- 172.21.15.88:0/1223098751 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 124047631 0 0) 0x7f7dd80b2c40 con 0x7f7dd80a9750 2024-03-29T10:50:02.272 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:02.289+0000 7f7dc6ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:50:02.272 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:02.289+0000 7f7dc6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:50:02.273 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:02.289+0000 7f7dc6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:50:02.273 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:02.289+0000 7f7dc6ffd640 10 monclient(hunting): my global_id is 4566 2024-03-29T10:50:02.273 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:02.289+0000 7f7dc6ffd640 1 -- 172.21.15.88:0/1223098751 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7db8004eb0 con 0x7f7dd80a9750 2024-03-29T10:50:02.273 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:02.290+0000 7f7dc6ffd640 1 -- 172.21.15.88:0/1223098751 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3538363853 0 0) 0x7f7db8004eb0 con 0x7f7dd80a9750 2024-03-29T10:50:02.273 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:02.290+0000 7f7dc6ffd640 1 -- 172.21.15.88:0/1223098751 --> v1:172.21.15.88:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7dd80b2c40 con 0x7f7dd80a9750 2024-03-29T10:50:02.273 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:02.290+0000 7f7dc6ffd640 1 -- 172.21.15.88:0/1223098751 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f7dcc001610 con 0x7f7dd80a9750 2024-03-29T10:50:02.273 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:02.290+0000 7f7dc6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:50:02.273 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:02.290+0000 7f7dc6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:50:02.273 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:02.290+0000 7f7dc6ffd640 10 monclient(hunting): dump: 2024-03-29T10:50:02.273 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:50:02.273 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:50:02.274 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:50:02.274 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:50:02.274 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:50:02.274 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:50:02.274 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:50:02.274 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:50:02.274 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:50:02.274 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:50:02.274 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:02.290+0000 7f7dc6ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:50:02.274 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:02.290+0000 7f7dc6ffd640 1 -- 172.21.15.88:0/1223098751 <== mon.0 v1:172.21.15.88:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2547871344 0 0) 0x7f7dd80b2c40 con 0x7f7dd80a9750 2024-03-29T10:50:02.274 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:02.290+0000 7f7dc6ffd640 1 -- 172.21.15.88:0/1223098751 >> v1:172.21.15.88:6790/0 conn(0x7f7dd8071c30 legacy=0x7f7dd81d6590 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:50:02.276 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:02.291+0000 7f7dc6ffd640 1 -- 172.21.15.88:0/1223098751 >> v1:172.21.15.196:6789/0 conn(0x7f7dd813e1e0 legacy=0x7f7dd81de8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:02.276 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:02.291+0000 7f7dc6ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:50:02.276 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:02.291+0000 7f7dc6ffd640 1 monclient: found mon.a 2024-03-29T10:50:02.276 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:02.291+0000 7f7dc6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:50:02.276 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:02.291+0000 7f7dc6ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-03-29T10:50:02.276 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:02.291+0000 7f7dc6ffd640 1 -- 172.21.15.88:0/1223098751 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7dd80b24f0 con 0x7f7dd80a9750 2024-03-29T10:50:02.276 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:02.291+0000 7f7dc6ffd640 10 monclient: _finish_auth 0 2024-03-29T10:50:02.276 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:02.291+0000 7f7dc6ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:50:02.276 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:02.291+0000 7f7dc6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:50:02.276 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:02.291+0000 7f7dc6ffd640 10 client.?.objecter ms_handle_connect 0x7f7dd813e1e0 2024-03-29T10:50:02.276 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:02.291+0000 7f7dc6ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:50:02.277 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:02.291+0000 7f7dc6ffd640 1 -- 172.21.15.88:0/1223098751 <== mon.0 v1:172.21.15.88:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7dcc004b00 con 0x7f7dd80a9750 2024-03-29T10:50:02.277 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:02.291+0000 7f7ddcf7b640 5 monclient: authenticate success, global_id 4566 2024-03-29T10:50:02.277 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:02.291+0000 7f7ddcf7b640 10 monclient: _renew_subs 2024-03-29T10:50:02.277 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:02.292+0000 7f7ddcf7b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-03-29T10:50:02.277 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:02.292+0000 7f7ddcf7b640 1 -- 172.21.15.88:0/1223098751 --> v1:172.21.15.88:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7dd8137180 con 0x7f7dd80a9750 2024-03-29T10:50:02.277 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:02.292+0000 7f7ddcf7b640 10 client.4566.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:50:02.277 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:02.292+0000 7f7ddcf7b640 10 monclient: _renew_subs 2024-03-29T10:50:02.277 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:02.292+0000 7f7ddcf7b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-03-29T10:50:02.277 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:02.292+0000 7f7ddcf7b640 1 -- 172.21.15.88:0/1223098751 --> v1:172.21.15.88:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7dd81d6ca0 con 0x7f7dd80a9750 2024-03-29T10:50:02.277 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:02.292+0000 7f7ddcf7b640 1 librados: init done 2024-03-29T10:50:02.277 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:02.292+0000 7f7dc6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:50:02.277 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:02.292+0000 7f7dc6ffd640 1 -- 172.21.15.88:0/1223098751 <== mon.0 v1:172.21.15.88:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f7dcc004f10 con 0x7f7dd80a9750 2024-03-29T10:50:02.277 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:02.292+0000 7f7dc6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:50:02.277 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:02.292+0000 7f7dc6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:50:02.277 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:02.292+0000 7f7dc6ffd640 10 monclient: dump: 2024-03-29T10:50:02.277 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:50:02.278 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:50:02.278 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:50:02.278 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:50:02.278 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:50:02.278 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:50:02.278 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:50:02.278 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:50:02.278 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:50:02.278 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:50:02.278 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:02.293+0000 7f7dd4d99640 4 set_mon_vals no callback set 2024-03-29T10:50:02.281 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:02.296+0000 7f7dc4ff9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:50:02.282 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:02.296+0000 7f7dc4ff9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:50:02.282 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:02.296+0000 7f7dc4ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-03-29T10:50:02.282 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:02.297+0000 7f7dc6ffd640 1 -- 172.21.15.88:0/1223098751 <== mon.0 v1:172.21.15.88:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4235629220 0 0) 0x7f7dcc01aee0 con 0x7f7dd80a9750 2024-03-29T10:50:02.282 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:02.298+0000 7f7dc6ffd640 1 -- 172.21.15.88:0/1223098751 <== mon.0 v1:172.21.15.88:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4257673582 0 0) 0x7f7dcc084b40 con 0x7f7dd80a9750 2024-03-29T10:50:02.282 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:02.298+0000 7f7dc6ffd640 10 client.4566.objecter ms_dispatch 0x7f7dd8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:50:02.282 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:02.298+0000 7f7dc6ffd640 3 client.4566.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:50:02.282 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:02.298+0000 7f7dc6ffd640 3 client.4566.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:50:02.282 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:02.298+0000 7f7dc6ffd640 20 client.4566.objecter dump_active .. 0 homeless 2024-03-29T10:50:02.282 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:02.296+0000 7f7dc4ff9640 1 -- 172.21.15.88:0/1223098751 --> v1:172.21.15.88:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7dd8075f10 con 0x7f7dd80a9750 2024-03-29T10:50:02.282 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:02.298+0000 7f7dd659c640 10 monclient: get_auth_request con 0x7f7db806fd90 auth_method 0 2024-03-29T10:50:02.282 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:02.299+0000 7f7dc6ffd640 10 client.4566.objecter ms_handle_connect 0x7f7db806fd90 2024-03-29T10:50:02.291 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:02.309+0000 7f7dc6ffd640 1 -- 172.21.15.88:0/1223098751 <== mon.0 v1:172.21.15.88:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7dcc050420 con 0x7f7dd80a9750 2024-03-29T10:50:02.291 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:02.309+0000 7f7dc6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:50:02.291 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:02.309+0000 7f7dc6ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:50:02.624 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:02.642+0000 7f7dc4ff9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:50:02.624 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:02.642+0000 7f7dc4ff9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:50:02.624 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:02.642+0000 7f7dc4ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-03-29T10:50:02.624 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:02.642+0000 7f7dc4ff9640 1 -- 172.21.15.88:0/1223098751 --> v1:172.21.15.88:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7dd8071a10 con 0x7f7dd80a9750 2024-03-29T10:50:02.625 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:02.643+0000 7f7dc6ffd640 1 -- 172.21.15.88:0/1223098751 <== mon.0 v1:172.21.15.88:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 344506374) 0x7f7dcc052350 con 0x7f7dd80a9750 2024-03-29T10:50:02.625 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:02.643+0000 7f7dc6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:50:02.625 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:02.643+0000 7f7dc6ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:50:02.625 INFO:teuthology.misc.health.smithi088.stdout: 2024-03-29T10:50:02.625 INFO:teuthology.misc.health.smithi088.stdout:{"epoch":10,"fsid":"869f5188-f92b-42dd-a46d-8c646e0388a9","created":"2024-03-29T10:40:40.285087+0000","modified":"2024-03-29T10:40:42.536230+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:40:42.536230+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a26c0c1c-8de0-438c-a3f6-3df10ea61574","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5de24915-8dec-49b6-8b16-f3170197c591","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ac054af3-d589-4565-9644-e6af75975c53","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3ec4f045-c714-4a22-bd1a-e07330b45ab3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"cddb9867-8e30-4951-9b13-3d1e78b9c45a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dc6d9911-785d-432f-8b53-56a1ccab302f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"77c88437-b587-496e-8b4d-1824ea8b5305","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ce963d96-5a8e-44c3-99de-c0dbebe234ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:50:02.629 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:02.646+0000 7f7dc4ff9640 10 librados: watch_flush enter 2024-03-29T10:50:02.629 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:02.647+0000 7f7dc4ff9640 10 librados: watch_flush exit 2024-03-29T10:50:02.629 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:02.647+0000 7f7dc4ff9640 20 client.4566.objecter shutdown clearing up homeless session... 2024-03-29T10:50:02.629 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:02.647+0000 7f7dc4ff9640 10 client.4566.objecter successfully canceled tick 2024-03-29T10:50:02.629 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:02.647+0000 7f7dc4ff9640 1 -- 172.21.15.88:0/1223098751 >> v1:172.21.15.88:6816/1752861349 conn(0x7f7db806fd90 legacy=0x7f7db8072250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:02.629 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:02.647+0000 7f7dc4ff9640 10 monclient: shutdown 2024-03-29T10:50:02.629 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:02.647+0000 7f7dc4ff9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:50:02.630 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:02.647+0000 7f7dc4ff9640 1 -- 172.21.15.88:0/1223098751 >> v1:172.21.15.88:6789/0 conn(0x7f7dd80a9750 legacy=0x7f7dd81307d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:02.630 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:02.647+0000 7f7dc4ff9640 1 -- 172.21.15.88:0/1223098751 shutdown_connections 2024-03-29T10:50:02.630 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:02.647+0000 7f7dc4ff9640 1 -- 172.21.15.88:0/1223098751 >> 172.21.15.88:0/1223098751 conn(0x7f7dd806b660 msgr2=0x7f7dd8139820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:50:02.630 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:02.648+0000 7f7dc4ff9640 1 -- 172.21.15.88:0/1223098751 shutdown_connections 2024-03-29T10:50:02.630 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:02.648+0000 7f7dc4ff9640 1 -- 172.21.15.88:0/1223098751 wait complete. 2024-03-29T10:50:02.630 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:02.648+0000 7f7dc4ff9640 1 librados: shutdown 2024-03-29T10:50:02.648 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:50:09.456 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:50:09.604 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:09.622+0000 7f11bd9e9640 10 monclient: get_monmap_and_config 2024-03-29T10:50:09.605 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:09.622+0000 7f11bd9e9640 10 monclient: build_initial_monmap 2024-03-29T10:50:09.605 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:09.622+0000 7f11bd9e9640 10 monclient: monmap: 2024-03-29T10:50:09.605 INFO:teuthology.misc.health.smithi088.stderr:epoch 0 2024-03-29T10:50:09.605 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:50:09.605 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:50:09.623558+0000 2024-03-29T10:50:09.605 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:50:09.623558+0000 2024-03-29T10:50:09.605 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 0 (unknown) 2024-03-29T10:50:09.605 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:50:09.605 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-03-29T10:50:09.605 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-29T10:50:09.605 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-b 2024-03-29T10:50:09.605 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:50:09.608 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:09.625+0000 7f11bd9e9640 1 Processor -- start 2024-03-29T10:50:09.608 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:09.626+0000 7f11bd9e9640 1 -- start start 2024-03-29T10:50:09.608 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:09.626+0000 7f11bd9e9640 10 monclient: init 2024-03-29T10:50:09.609 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:09.627+0000 7f11bd9e9640 10 monclient: _reopen_session rank -1 2024-03-29T10:50:09.609 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:09.627+0000 7f11bd9e9640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:50:09.609 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:09.627+0000 7f11bd9e9640 10 monclient(hunting): picked mon.noname-b con 0x7f11b80b0860 addr v1:172.21.15.88:6790/0 2024-03-29T10:50:09.609 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:09.627+0000 7f11bd9e9640 10 monclient(hunting): picked mon.noname-a con 0x7f11b813ef30 addr v1:172.21.15.88:6789/0 2024-03-29T10:50:09.609 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:09.627+0000 7f11bd9e9640 10 monclient(hunting): picked mon.noname-c con 0x7f11b80af8e0 addr v1:172.21.15.196:6789/0 2024-03-29T10:50:09.609 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:09.627+0000 7f11bd9e9640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11b80b0c40 con 0x7f11b813ef30 2024-03-29T10:50:09.609 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:09.627+0000 7f11bd9e9640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11b80aed10 con 0x7f11b80af8e0 2024-03-29T10:50:09.609 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:09.627+0000 7f11bd9e9640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11b80afd50 con 0x7f11b80b0860 2024-03-29T10:50:09.610 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:09.627+0000 7f11bd9e9640 10 monclient(hunting): _renew_subs 2024-03-29T10:50:09.610 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:09.627+0000 7f11bd9e9640 10 monclient(hunting): authenticate will time out at 1331.863401s 2024-03-29T10:50:09.610 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:09.627+0000 7f11b6ffd640 1 --1- >> v1:172.21.15.88:6790/0 conn(0x7f11b80b0860 0x7f11b813e820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6790/0 says I am v1:172.21.15.88:60634/0 (socket says 172.21.15.88:60634) 2024-03-29T10:50:09.610 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:09.627+0000 7f11b6ffd640 1 -- 172.21.15.88:0/4003609081 learned_addr learned my addr 172.21.15.88:0/4003609081 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-29T10:50:09.612 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:09.630+0000 7f11b5ffb640 1 -- 172.21.15.88:0/4003609081 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1852184760 0 0) 0x7f11b80afd50 con 0x7f11b80b0860 2024-03-29T10:50:09.612 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:09.630+0000 7f11b5ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:50:09.612 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:09.630+0000 7f11b5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:50:09.612 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:09.630+0000 7f11b5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:50:09.612 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:09.630+0000 7f11b5ffb640 10 monclient(hunting): my global_id is 4598 2024-03-29T10:50:09.612 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:09.630+0000 7f11b5ffb640 1 -- 172.21.15.88:0/4003609081 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f11a0003520 con 0x7f11b80b0860 2024-03-29T10:50:09.613 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:09.630+0000 7f11b5ffb640 1 -- 172.21.15.88:0/4003609081 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3342826447 0 0) 0x7f11a0003520 con 0x7f11b80b0860 2024-03-29T10:50:09.613 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:09.630+0000 7f11b5ffb640 1 -- 172.21.15.88:0/4003609081 >> v1:172.21.15.196:6789/0 conn(0x7f11b80af8e0 legacy=0x7f11b8149a70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:50:09.616 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:09.633+0000 7f11b5ffb640 1 -- 172.21.15.88:0/4003609081 >> v1:172.21.15.88:6789/0 conn(0x7f11b813ef30 legacy=0x7f11b80af1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:09.616 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:09.633+0000 7f11b5ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:50:09.616 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:09.633+0000 7f11b5ffb640 1 monclient: found mon.noname-b 2024-03-29T10:50:09.616 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:09.634+0000 7f11b5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:50:09.616 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:09.634+0000 7f11b5ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.88:6790/0 2024-03-29T10:50:09.616 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:09.634+0000 7f11b5ffb640 1 -- 172.21.15.88:0/4003609081 --> v1:172.21.15.88:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11b80ab1b0 con 0x7f11b80b0860 2024-03-29T10:50:09.616 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:09.634+0000 7f11b5ffb640 10 monclient: _finish_auth 0 2024-03-29T10:50:09.616 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:09.634+0000 7f11b5ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:50:09.616 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:09.634+0000 7f11b5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:50:09.616 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:09.634+0000 7f11b5ffb640 1 -- 172.21.15.88:0/4003609081 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f11b0001a10 con 0x7f11b80b0860 2024-03-29T10:50:09.617 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:09.634+0000 7f11bd9e9640 5 monclient: authenticate success, global_id 4598 2024-03-29T10:50:09.617 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:09.634+0000 7f11bd9e9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:50:09.617 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:09.634+0000 7f11b5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:50:09.617 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:09.634+0000 7f11b5ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:50:09.617 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:09.634+0000 7f11b5ffb640 10 monclient: dump: 2024-03-29T10:50:09.617 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:50:09.617 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:50:09.617 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:50:09.617 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:50:09.617 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:50:09.617 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:50:09.617 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:50:09.617 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:50:09.618 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:50:09.618 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:50:09.618 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:09.634+0000 7f11b5ffb640 1 -- 172.21.15.88:0/4003609081 <== mon.2 v1:172.21.15.88:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f11b0001ce0 con 0x7f11b80b0860 2024-03-29T10:50:09.618 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:09.634+0000 7f11b5ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:50:09.618 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:09.634+0000 7f11b5ffb640 1 -- 172.21.15.88:0/4003609081 <== mon.2 v1:172.21.15.88:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f11b0002070 con 0x7f11b80b0860 2024-03-29T10:50:09.618 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:09.634+0000 7f11b5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:50:09.618 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:09.634+0000 7f11b5ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:50:09.618 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:09.634+0000 7f11b5ffb640 10 monclient: dump: 2024-03-29T10:50:09.618 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:50:09.618 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:50:09.618 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:50:09.618 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:50:09.618 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:50:09.619 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:50:09.619 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:50:09.619 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:50:09.619 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:50:09.619 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:50:09.619 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:09.634+0000 7f11bd9e9640 10 monclient: get_monmap_and_config success 2024-03-29T10:50:09.619 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:09.634+0000 7f11bd9e9640 4 set_mon_vals no callback set 2024-03-29T10:50:09.620 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:09.635+0000 7f11bd9e9640 10 monclient: shutdown 2024-03-29T10:50:09.620 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:09.635+0000 7f11bd9e9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:50:09.620 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:09.635+0000 7f11bd9e9640 1 -- 172.21.15.88:0/4003609081 >> v1:172.21.15.88:6790/0 conn(0x7f11b80b0860 legacy=0x7f11b813e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:09.620 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:09.636+0000 7f11bd9e9640 1 -- 172.21.15.88:0/4003609081 shutdown_connections 2024-03-29T10:50:09.620 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:09.637+0000 7f11bd9e9640 1 -- 172.21.15.88:0/4003609081 >> 172.21.15.88:0/4003609081 conn(0x7f11b806b660 msgr2=0x7f11b806ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:50:09.621 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:09.638+0000 7f11bd9e9640 1 -- 172.21.15.88:0/4003609081 shutdown_connections 2024-03-29T10:50:09.621 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:09.638+0000 7f11bd9e9640 1 -- 172.21.15.88:0/4003609081 wait complete. 2024-03-29T10:50:09.621 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:09.640+0000 7f11bd9e9640 10 monclient: build_initial_monmap 2024-03-29T10:50:09.621 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:09.640+0000 7f11bd9e9640 10 monclient: monmap: 2024-03-29T10:50:09.621 INFO:teuthology.misc.health.smithi088.stderr:epoch 0 2024-03-29T10:50:09.621 INFO:teuthology.misc.health.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:50:09.622 INFO:teuthology.misc.health.smithi088.stderr:last_changed 0.000000 2024-03-29T10:50:09.622 INFO:teuthology.misc.health.smithi088.stderr:created 0.000000 2024-03-29T10:50:09.622 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 0 (unknown) 2024-03-29T10:50:09.622 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:50:09.622 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-03-29T10:50:09.622 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-29T10:50:09.622 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-c 2024-03-29T10:50:09.622 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:50:09.623 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:09.640+0000 7f11bd9e9640 1 librados: starting msgr at 2024-03-29T10:50:09.623 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:09.640+0000 7f11bd9e9640 1 librados: starting objecter 2024-03-29T10:50:09.623 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:09.640+0000 7f11bd9e9640 1 Processor -- start 2024-03-29T10:50:09.623 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:09.640+0000 7f11bd9e9640 1 -- start start 2024-03-29T10:50:09.623 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:09.640+0000 7f11bd9e9640 1 librados: setting wanted keys 2024-03-29T10:50:09.623 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:09.640+0000 7f11bd9e9640 1 librados: calling monclient init 2024-03-29T10:50:09.623 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:09.640+0000 7f11bd9e9640 10 monclient: init 2024-03-29T10:50:09.623 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:09.641+0000 7f11bd9e9640 10 monclient: _reopen_session rank -1 2024-03-29T10:50:09.623 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:09.641+0000 7f11bd9e9640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:50:09.623 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:09.641+0000 7f11bd9e9640 10 monclient(hunting): picked mon.noname-c con 0x7f11b80af8e0 addr v1:172.21.15.88:6790/0 2024-03-29T10:50:09.624 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:09.641+0000 7f11bd9e9640 10 monclient(hunting): picked mon.noname-a con 0x7f11b80b0860 addr v1:172.21.15.88:6789/0 2024-03-29T10:50:09.624 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:09.641+0000 7f11bd9e9640 10 monclient(hunting): picked mon.noname-b con 0x7f11b813ef30 addr v1:172.21.15.196:6789/0 2024-03-29T10:50:09.624 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:09.641+0000 7f11bd9e9640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11b806ed80 con 0x7f11b80b0860 2024-03-29T10:50:09.624 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:09.641+0000 7f11bd9e9640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11b806f060 con 0x7f11b813ef30 2024-03-29T10:50:09.624 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:09.641+0000 7f11bd9e9640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11b8071b30 con 0x7f11b80af8e0 2024-03-29T10:50:09.624 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:09.641+0000 7f11bd9e9640 10 monclient(hunting): _renew_subs 2024-03-29T10:50:09.624 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:09.641+0000 7f11bd9e9640 10 monclient(hunting): authenticate will time out at 1331.877631s 2024-03-29T10:50:09.626 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:09.644+0000 7f11b6ffd640 1 --1- >> v1:172.21.15.88:6790/0 conn(0x7f11b80af8e0 0x7f11b81deea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6790/0 says I am v1:172.21.15.88:52988/0 (socket says 172.21.15.88:52988) 2024-03-29T10:50:09.626 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:09.644+0000 7f11b6ffd640 1 -- 172.21.15.88:0/1078639655 learned_addr learned my addr 172.21.15.88:0/1078639655 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-29T10:50:09.627 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:09.645+0000 7f119ffff640 10 client.?.objecter ms_handle_connect 0x7f11b80af8e0 2024-03-29T10:50:09.627 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:09.645+0000 7f119ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:50:09.627 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:09.645+0000 7f119ffff640 1 -- 172.21.15.88:0/1078639655 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2507101736 0 0) 0x7f11b8071b30 con 0x7f11b80af8e0 2024-03-29T10:50:09.627 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:09.645+0000 7f119ffff640 10 monclient(hunting): _negotiate 2024-03-29T10:50:09.627 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:09.645+0000 7f119ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:50:09.627 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:09.645+0000 7f119ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:50:09.627 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:09.645+0000 7f119ffff640 10 monclient(hunting): my global_id is 4601 2024-03-29T10:50:09.627 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:09.645+0000 7f119ffff640 1 -- 172.21.15.88:0/1078639655 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f11a4004eb0 con 0x7f11b80af8e0 2024-03-29T10:50:09.630 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:09.646+0000 7f119ffff640 10 client.?.objecter ms_handle_connect 0x7f11b813ef30 2024-03-29T10:50:09.630 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:09.646+0000 7f119ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:50:09.630 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:09.646+0000 7f119ffff640 1 -- 172.21.15.88:0/1078639655 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3146823093 0 0) 0x7f11b806f060 con 0x7f11b813ef30 2024-03-29T10:50:09.631 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:09.646+0000 7f119ffff640 10 monclient(hunting): _negotiate 2024-03-29T10:50:09.631 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:09.646+0000 7f119ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:50:09.631 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:09.646+0000 7f119ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:50:09.631 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:09.646+0000 7f119ffff640 10 monclient(hunting): my global_id is 4549 2024-03-29T10:50:09.631 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:09.646+0000 7f119ffff640 1 -- 172.21.15.88:0/1078639655 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f11b8071b30 con 0x7f11b813ef30 2024-03-29T10:50:09.631 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:09.646+0000 7f119ffff640 10 client.?.objecter ms_handle_connect 0x7f11b80b0860 2024-03-29T10:50:09.631 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:09.646+0000 7f119ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:50:09.631 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:09.646+0000 7f119ffff640 1 -- 172.21.15.88:0/1078639655 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 338810416 0 0) 0x7f11b806ed80 con 0x7f11b80b0860 2024-03-29T10:50:09.631 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:09.647+0000 7f119ffff640 10 monclient(hunting): _negotiate 2024-03-29T10:50:09.631 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:09.647+0000 7f119ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:50:09.631 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:09.647+0000 7f119ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:50:09.632 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:09.647+0000 7f119ffff640 10 monclient(hunting): my global_id is 4572 2024-03-29T10:50:09.632 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:09.647+0000 7f119ffff640 1 -- 172.21.15.88:0/1078639655 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f11b806f060 con 0x7f11b80b0860 2024-03-29T10:50:09.632 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:09.647+0000 7f119ffff640 1 -- 172.21.15.88:0/1078639655 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 353099095 0 0) 0x7f11b8071b30 con 0x7f11b813ef30 2024-03-29T10:50:09.632 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:09.647+0000 7f119ffff640 1 -- 172.21.15.88:0/1078639655 --> v1:172.21.15.196:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f11b806ed80 con 0x7f11b813ef30 2024-03-29T10:50:09.632 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:09.647+0000 7f119ffff640 1 -- 172.21.15.88:0/1078639655 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f11a80017b0 con 0x7f11b813ef30 2024-03-29T10:50:09.632 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:09.647+0000 7f119ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:50:09.632 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:09.647+0000 7f119ffff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:50:09.632 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:09.647+0000 7f119ffff640 10 monclient(hunting): dump: 2024-03-29T10:50:09.632 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:50:09.632 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:50:09.632 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:50:09.632 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:50:09.632 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:50:09.632 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:50:09.633 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:50:09.633 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:50:09.633 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:50:09.633 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:50:09.633 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:09.647+0000 7f119ffff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:50:09.633 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:09.647+0000 7f119ffff640 1 -- 172.21.15.88:0/1078639655 <== mon.1 v1:172.21.15.196:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3862190541 0 0) 0x7f11b806ed80 con 0x7f11b813ef30 2024-03-29T10:50:09.633 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:09.647+0000 7f119ffff640 1 -- 172.21.15.88:0/1078639655 >> v1:172.21.15.88:6790/0 conn(0x7f11b80af8e0 legacy=0x7f11b81deea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:09.633 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:09.647+0000 7f119ffff640 1 -- 172.21.15.88:0/1078639655 >> v1:172.21.15.88:6789/0 conn(0x7f11b80b0860 legacy=0x7f11b8135580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:09.633 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:09.647+0000 7f119ffff640 10 monclient: _finish_hunting 0 2024-03-29T10:50:09.633 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:09.647+0000 7f119ffff640 1 monclient: found mon.b 2024-03-29T10:50:09.633 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:09.647+0000 7f119ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:50:09.633 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:09.647+0000 7f119ffff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T10:50:09.633 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:09.647+0000 7f119ffff640 1 -- 172.21.15.88:0/1078639655 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11b806dc40 con 0x7f11b813ef30 2024-03-29T10:50:09.633 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:09.647+0000 7f119ffff640 10 monclient: _finish_auth 0 2024-03-29T10:50:09.634 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:09.647+0000 7f119ffff640 10 monclient: _check_auth_tickets 2024-03-29T10:50:09.634 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:09.647+0000 7f119ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:50:09.634 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:09.647+0000 7f11bd9e9640 5 monclient: authenticate success, global_id 4549 2024-03-29T10:50:09.634 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:09.647+0000 7f11bd9e9640 10 monclient: _renew_subs 2024-03-29T10:50:09.634 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:09.647+0000 7f11bd9e9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T10:50:09.634 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:09.647+0000 7f11bd9e9640 1 -- 172.21.15.88:0/1078639655 --> v1:172.21.15.196:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11b806de30 con 0x7f11b813ef30 2024-03-29T10:50:09.634 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:09.647+0000 7f11bd9e9640 10 client.4549.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:50:09.634 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:09.647+0000 7f11bd9e9640 10 monclient: _renew_subs 2024-03-29T10:50:09.634 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:09.647+0000 7f11bd9e9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T10:50:09.634 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:09.647+0000 7f11bd9e9640 1 -- 172.21.15.88:0/1078639655 --> v1:172.21.15.196:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f11b8138820 con 0x7f11b813ef30 2024-03-29T10:50:09.634 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:09.647+0000 7f11bd9e9640 1 librados: init done 2024-03-29T10:50:09.634 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:09.647+0000 7f119ffff640 1 -- 172.21.15.88:0/1078639655 <== mon.1 v1:172.21.15.196:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f11a8001cc0 con 0x7f11b813ef30 2024-03-29T10:50:09.634 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:09.647+0000 7f119ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:50:09.634 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:09.647+0000 7f119ffff640 1 -- 172.21.15.88:0/1078639655 <== mon.1 v1:172.21.15.196:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f11a8003830 con 0x7f11b813ef30 2024-03-29T10:50:09.635 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:09.647+0000 7f119ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:50:09.635 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:09.647+0000 7f119ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:50:09.635 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:09.647+0000 7f119ffff640 10 monclient: dump: 2024-03-29T10:50:09.635 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:50:09.635 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:50:09.635 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:50:09.635 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:50:09.635 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:50:09.635 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:50:09.635 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:50:09.635 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:50:09.635 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:50:09.635 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:50:09.635 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:09.647+0000 7f11b4ff9640 4 set_mon_vals no callback set 2024-03-29T10:50:09.636 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:09.649+0000 7f11bd9e9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:50:09.636 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:09.649+0000 7f11bd9e9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:50:09.637 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:09.649+0000 7f11bd9e9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T10:50:09.637 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:09.649+0000 7f11bd9e9640 1 -- 172.21.15.88:0/1078639655 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1198005080 con 0x7f11b813ef30 2024-03-29T10:50:09.637 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:09.651+0000 7f119ffff640 1 -- 172.21.15.88:0/1078639655 <== mon.1 v1:172.21.15.196:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4235629220 0 0) 0x7f11a8019800 con 0x7f11b813ef30 2024-03-29T10:50:09.637 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:09.652+0000 7f119ffff640 1 -- 172.21.15.88:0/1078639655 <== mon.1 v1:172.21.15.196:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4257673582 0 0) 0x7f11a8083710 con 0x7f11b813ef30 2024-03-29T10:50:09.637 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:09.652+0000 7f119ffff640 10 client.4549.objecter ms_dispatch 0x7f11b8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:50:09.637 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:09.652+0000 7f119ffff640 3 client.4549.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:50:09.637 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:09.652+0000 7f119ffff640 3 client.4549.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:50:09.637 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:09.652+0000 7f11b6ffd640 10 monclient: get_auth_request con 0x7f11a407ee10 auth_method 0 2024-03-29T10:50:09.637 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:09.652+0000 7f119ffff640 20 client.4549.objecter dump_active .. 0 homeless 2024-03-29T10:50:09.637 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:09.653+0000 7f119ffff640 10 client.4549.objecter ms_handle_connect 0x7f11a407ee10 2024-03-29T10:50:09.641 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:09.659+0000 7f119ffff640 1 -- 172.21.15.88:0/1078639655 <== mon.1 v1:172.21.15.196:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f11a804eff0 con 0x7f11b813ef30 2024-03-29T10:50:09.641 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:09.659+0000 7f119ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:50:09.641 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:09.659+0000 7f119ffff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:50:09.961 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:09.978+0000 7f11bd9e9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:50:09.961 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:09.978+0000 7f11bd9e9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:50:09.961 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:09.978+0000 7f11bd9e9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T10:50:09.961 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:09.978+0000 7f11bd9e9640 1 -- 172.21.15.88:0/1078639655 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1198005330 con 0x7f11b813ef30 2024-03-29T10:50:09.961 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:09.979+0000 7f119ffff640 1 -- 172.21.15.88:0/1078639655 <== mon.1 v1:172.21.15.196:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 344506374) 0x7f11a8050f20 con 0x7f11b813ef30 2024-03-29T10:50:09.961 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:09.979+0000 7f119ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:50:09.961 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:09.979+0000 7f119ffff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:50:09.962 INFO:teuthology.misc.health.smithi088.stdout: 2024-03-29T10:50:09.962 INFO:teuthology.misc.health.smithi088.stdout:{"epoch":10,"fsid":"869f5188-f92b-42dd-a46d-8c646e0388a9","created":"2024-03-29T10:40:40.285087+0000","modified":"2024-03-29T10:40:42.536230+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:40:42.536230+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a26c0c1c-8de0-438c-a3f6-3df10ea61574","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5de24915-8dec-49b6-8b16-f3170197c591","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ac054af3-d589-4565-9644-e6af75975c53","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3ec4f045-c714-4a22-bd1a-e07330b45ab3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"cddb9867-8e30-4951-9b13-3d1e78b9c45a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dc6d9911-785d-432f-8b53-56a1ccab302f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"77c88437-b587-496e-8b4d-1824ea8b5305","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ce963d96-5a8e-44c3-99de-c0dbebe234ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:50:09.967 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:09.983+0000 7f11bd9e9640 10 librados: watch_flush enter 2024-03-29T10:50:09.967 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:09.983+0000 7f11bd9e9640 10 librados: watch_flush exit 2024-03-29T10:50:09.967 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:09.983+0000 7f11bd9e9640 20 client.4549.objecter shutdown clearing up homeless session... 2024-03-29T10:50:09.967 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:09.983+0000 7f11bd9e9640 10 client.4549.objecter successfully canceled tick 2024-03-29T10:50:09.967 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:09.983+0000 7f11bd9e9640 1 -- 172.21.15.88:0/1078639655 >> v1:172.21.15.88:6816/1752861349 conn(0x7f11a407ee10 legacy=0x7f11a40812d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:09.967 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:09.983+0000 7f11bd9e9640 10 monclient: shutdown 2024-03-29T10:50:09.967 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:09.983+0000 7f11bd9e9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:50:09.967 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:09.983+0000 7f11bd9e9640 1 -- 172.21.15.88:0/1078639655 >> v1:172.21.15.196:6789/0 conn(0x7f11b813ef30 legacy=0x7f11b81e6170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:09.967 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:09.983+0000 7f11bd9e9640 1 -- 172.21.15.88:0/1078639655 shutdown_connections 2024-03-29T10:50:09.967 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:09.983+0000 7f11bd9e9640 1 -- 172.21.15.88:0/1078639655 >> 172.21.15.88:0/1078639655 conn(0x7f11b806b660 msgr2=0x7f11b813e5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:50:09.967 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:09.984+0000 7f11bd9e9640 1 -- 172.21.15.88:0/1078639655 shutdown_connections 2024-03-29T10:50:09.967 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:09.984+0000 7f11bd9e9640 1 -- 172.21.15.88:0/1078639655 wait complete. 2024-03-29T10:50:09.968 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:09.984+0000 7f11bd9e9640 1 librados: shutdown 2024-03-29T10:50:09.984 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:50:16.792 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:50:16.946 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:16.964+0000 7f35c84c9640 10 monclient: get_monmap_and_config 2024-03-29T10:50:16.946 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:16.964+0000 7f35c84c9640 10 monclient: build_initial_monmap 2024-03-29T10:50:16.946 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:16.964+0000 7f35c84c9640 10 monclient: monmap: 2024-03-29T10:50:16.946 INFO:teuthology.misc.health.smithi088.stderr:epoch 0 2024-03-29T10:50:16.946 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:50:16.946 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:50:16.965336+0000 2024-03-29T10:50:16.946 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:50:16.965336+0000 2024-03-29T10:50:16.946 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 0 (unknown) 2024-03-29T10:50:16.946 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:50:16.947 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-03-29T10:50:16.947 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-29T10:50:16.947 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-b 2024-03-29T10:50:16.947 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:50:16.947 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:16.965+0000 7f35c84c9640 1 Processor -- start 2024-03-29T10:50:16.947 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:16.965+0000 7f35c84c9640 1 -- start start 2024-03-29T10:50:16.947 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:16.965+0000 7f35c84c9640 10 monclient: init 2024-03-29T10:50:16.949 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:16.966+0000 7f35c84c9640 10 monclient: _reopen_session rank -1 2024-03-29T10:50:16.949 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:16.966+0000 7f35c84c9640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:50:16.949 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:16.966+0000 7f35c84c9640 10 monclient(hunting): picked mon.noname-b con 0x7f35c006c250 addr v1:172.21.15.88:6790/0 2024-03-29T10:50:16.949 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:16.966+0000 7f35c84c9640 10 monclient(hunting): picked mon.noname-a con 0x7f35c00ae650 addr v1:172.21.15.88:6789/0 2024-03-29T10:50:16.949 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:16.966+0000 7f35c84c9640 10 monclient(hunting): picked mon.noname-c con 0x7f35c013f8c0 addr v1:172.21.15.196:6789/0 2024-03-29T10:50:16.949 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:16.966+0000 7f35c84c9640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35c00adf50 con 0x7f35c00ae650 2024-03-29T10:50:16.949 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:16.966+0000 7f35c84c9640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35c00ae230 con 0x7f35c013f8c0 2024-03-29T10:50:16.949 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:16.966+0000 7f35c84c9640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35c013fd50 con 0x7f35c006c250 2024-03-29T10:50:16.949 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:16.966+0000 7f35c84c9640 10 monclient(hunting): _renew_subs 2024-03-29T10:50:16.949 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:16.966+0000 7f35c84c9640 10 monclient(hunting): authenticate will time out at 1339.202882s 2024-03-29T10:50:16.950 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:16.967+0000 7f35c6265640 1 --1- >> v1:172.21.15.88:6790/0 conn(0x7f35c006c250 0x7f35c01427b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6790/0 says I am v1:172.21.15.88:59152/0 (socket says 172.21.15.88:59152) 2024-03-29T10:50:16.950 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:16.967+0000 7f35c6265640 1 -- 172.21.15.88:0/1983303677 learned_addr learned my addr 172.21.15.88:0/1983303677 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-29T10:50:16.950 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:16.968+0000 7f35c5263640 1 -- 172.21.15.88:0/1983303677 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3253126133 0 0) 0x7f35c00ae230 con 0x7f35c013f8c0 2024-03-29T10:50:16.950 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:16.968+0000 7f35c5263640 10 monclient(hunting): _negotiate 2024-03-29T10:50:16.950 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:16.968+0000 7f35c5263640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:50:16.950 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:16.968+0000 7f35c5263640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:50:16.950 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:16.968+0000 7f35c5263640 10 monclient(hunting): my global_id is 4552 2024-03-29T10:50:16.951 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:16.968+0000 7f35c5263640 1 -- 172.21.15.88:0/1983303677 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f35ac003520 con 0x7f35c013f8c0 2024-03-29T10:50:16.951 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:16.968+0000 7f35c5263640 1 -- 172.21.15.88:0/1983303677 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 410655242 0 0) 0x7f35c013fd50 con 0x7f35c006c250 2024-03-29T10:50:16.951 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:16.968+0000 7f35c5263640 10 monclient(hunting): _negotiate 2024-03-29T10:50:16.951 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:16.968+0000 7f35c5263640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:50:16.951 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:16.968+0000 7f35c5263640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:50:16.951 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:16.968+0000 7f35c5263640 10 monclient(hunting): my global_id is 4604 2024-03-29T10:50:16.951 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:16.968+0000 7f35c5263640 1 -- 172.21.15.88:0/1983303677 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f35c00ae230 con 0x7f35c006c250 2024-03-29T10:50:16.951 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:16.969+0000 7f35c5263640 1 -- 172.21.15.88:0/1983303677 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3757709857 0 0) 0x7f35ac003520 con 0x7f35c013f8c0 2024-03-29T10:50:16.951 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:16.969+0000 7f35c5263640 1 -- 172.21.15.88:0/1983303677 >> v1:172.21.15.88:6790/0 conn(0x7f35c006c250 legacy=0x7f35c01427b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:16.951 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:16.969+0000 7f35c5263640 1 -- 172.21.15.88:0/1983303677 >> v1:172.21.15.88:6789/0 conn(0x7f35c00ae650 legacy=0x7f35c013f1b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:50:16.953 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:16.970+0000 7f35c5263640 10 monclient: _finish_hunting 0 2024-03-29T10:50:16.953 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:16.970+0000 7f35c5263640 1 monclient: found mon.noname-c 2024-03-29T10:50:16.953 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:16.970+0000 7f35c5263640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:50:16.953 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:16.970+0000 7f35c5263640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.196:6789/0 2024-03-29T10:50:16.953 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:16.970+0000 7f35c5263640 1 -- 172.21.15.88:0/1983303677 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35c006db20 con 0x7f35c013f8c0 2024-03-29T10:50:16.953 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:16.970+0000 7f35c5263640 10 monclient: _finish_auth 0 2024-03-29T10:50:16.953 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:16.970+0000 7f35c5263640 10 monclient: _check_auth_tickets 2024-03-29T10:50:16.953 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:16.970+0000 7f35c5263640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:50:16.953 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:16.970+0000 7f35c84c9640 5 monclient: authenticate success, global_id 4552 2024-03-29T10:50:16.953 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:16.970+0000 7f35c84c9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:50:16.953 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:16.970+0000 7f35c5263640 1 -- 172.21.15.88:0/1983303677 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f35bc001690 con 0x7f35c013f8c0 2024-03-29T10:50:16.953 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:16.970+0000 7f35c5263640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:50:16.953 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:16.970+0000 7f35c5263640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:50:16.954 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:16.970+0000 7f35c5263640 10 monclient: dump: 2024-03-29T10:50:16.954 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:50:16.954 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:50:16.954 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:50:16.954 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:50:16.954 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:50:16.954 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:50:16.954 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:50:16.954 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:50:16.954 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:50:16.954 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:50:16.954 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:16.970+0000 7f35c84c9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:50:16.954 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:16.970+0000 7f35c5263640 1 -- 172.21.15.88:0/1983303677 <== mon.1 v1:172.21.15.196:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f35bc001870 con 0x7f35c013f8c0 2024-03-29T10:50:16.954 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:16.970+0000 7f35c5263640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:50:16.955 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:16.970+0000 7f35c84c9640 10 monclient: get_monmap_and_config success 2024-03-29T10:50:16.955 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:16.970+0000 7f35c84c9640 4 set_mon_vals no callback set 2024-03-29T10:50:16.956 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:16.971+0000 7f35c84c9640 10 monclient: shutdown 2024-03-29T10:50:16.956 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:16.971+0000 7f35c84c9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:50:16.956 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:16.971+0000 7f35c84c9640 1 -- 172.21.15.88:0/1983303677 >> v1:172.21.15.196:6789/0 conn(0x7f35c013f8c0 legacy=0x7f35c014db30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:16.956 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:16.971+0000 7f35c5263640 1 -- 172.21.15.88:0/1983303677 <== mon.1 v1:172.21.15.196:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f35bc001c80 con 0x7f35c013f8c0 2024-03-29T10:50:16.956 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:16.971+0000 7f35c5263640 10 monclient: discarding stray monitor message mon_map magic: 0 v1 2024-03-29T10:50:16.956 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:16.972+0000 7f35c84c9640 1 -- 172.21.15.88:0/1983303677 shutdown_connections 2024-03-29T10:50:16.956 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:16.972+0000 7f35c84c9640 1 -- 172.21.15.88:0/1983303677 >> 172.21.15.88:0/1983303677 conn(0x7f35c0067610 msgr2=0x7f35c00b5a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:50:16.957 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:16.972+0000 7f35c84c9640 1 -- 172.21.15.88:0/1983303677 shutdown_connections 2024-03-29T10:50:16.957 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:16.972+0000 7f35c84c9640 1 -- 172.21.15.88:0/1983303677 wait complete. 2024-03-29T10:50:16.957 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:16.973+0000 7f35c84c9640 10 monclient: build_initial_monmap 2024-03-29T10:50:16.957 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:16.973+0000 7f35c84c9640 10 monclient: monmap: 2024-03-29T10:50:16.957 INFO:teuthology.misc.health.smithi088.stderr:epoch 0 2024-03-29T10:50:16.957 INFO:teuthology.misc.health.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:50:16.957 INFO:teuthology.misc.health.smithi088.stderr:last_changed 0.000000 2024-03-29T10:50:16.957 INFO:teuthology.misc.health.smithi088.stderr:created 0.000000 2024-03-29T10:50:16.957 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 0 (unknown) 2024-03-29T10:50:16.957 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:50:16.957 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-03-29T10:50:16.957 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-29T10:50:16.957 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-c 2024-03-29T10:50:16.957 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:50:16.958 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:16.973+0000 7f35c84c9640 1 librados: starting msgr at 2024-03-29T10:50:16.958 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:16.973+0000 7f35c84c9640 1 librados: starting objecter 2024-03-29T10:50:16.958 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:16.973+0000 7f35c84c9640 1 Processor -- start 2024-03-29T10:50:16.958 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:16.973+0000 7f35c84c9640 1 -- start start 2024-03-29T10:50:16.958 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:16.973+0000 7f35c84c9640 1 librados: setting wanted keys 2024-03-29T10:50:16.958 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:16.973+0000 7f35c84c9640 1 librados: calling monclient init 2024-03-29T10:50:16.958 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:16.973+0000 7f35c84c9640 10 monclient: init 2024-03-29T10:50:16.959 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:16.974+0000 7f35c84c9640 10 monclient: _reopen_session rank -1 2024-03-29T10:50:16.959 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:16.974+0000 7f35c84c9640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:50:16.959 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:16.974+0000 7f35c84c9640 10 monclient(hunting): picked mon.noname-c con 0x7f35c006c250 addr v1:172.21.15.88:6790/0 2024-03-29T10:50:16.959 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:16.974+0000 7f35c84c9640 10 monclient(hunting): picked mon.noname-a con 0x7f35c00ae650 addr v1:172.21.15.88:6789/0 2024-03-29T10:50:16.959 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:16.974+0000 7f35c84c9640 10 monclient(hunting): picked mon.noname-b con 0x7f35c013f8c0 addr v1:172.21.15.196:6789/0 2024-03-29T10:50:16.959 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:16.974+0000 7f35c84c9640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35c0143f60 con 0x7f35c00ae650 2024-03-29T10:50:16.959 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:16.975+0000 7f35c84c9640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35c0071fc0 con 0x7f35c013f8c0 2024-03-29T10:50:16.959 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:16.975+0000 7f35c84c9640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35c0133b70 con 0x7f35c006c250 2024-03-29T10:50:16.959 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:16.975+0000 7f35c84c9640 10 monclient(hunting): _renew_subs 2024-03-29T10:50:16.960 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:16.975+0000 7f35c84c9640 10 monclient(hunting): authenticate will time out at 1339.210912s 2024-03-29T10:50:16.960 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:16.975+0000 7f35c6a66640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f35c013f8c0 0x7f35c0133460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.88:45218/0 (socket says 172.21.15.88:45218) 2024-03-29T10:50:16.960 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:16.975+0000 7f35c6a66640 1 -- 172.21.15.88:0/2870213217 learned_addr learned my addr 172.21.15.88:0/2870213217 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-29T10:50:16.960 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:16.976+0000 7f35b6ffd640 10 client.?.objecter ms_handle_connect 0x7f35c006c250 2024-03-29T10:50:16.960 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:16.976+0000 7f35b6ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:50:16.960 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:16.976+0000 7f35b6ffd640 10 client.?.objecter ms_handle_connect 0x7f35c013f8c0 2024-03-29T10:50:16.960 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:16.976+0000 7f35b6ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:50:16.960 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:16.977+0000 7f35b6ffd640 1 -- 172.21.15.88:0/2870213217 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3932180882 0 0) 0x7f35c0133b70 con 0x7f35c006c250 2024-03-29T10:50:16.960 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:16.977+0000 7f35b6ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:50:16.960 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:16.977+0000 7f35b6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:50:16.960 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:16.977+0000 7f35b6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:50:16.960 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:16.977+0000 7f35b6ffd640 10 monclient(hunting): my global_id is 4607 2024-03-29T10:50:16.960 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:16.977+0000 7f35b6ffd640 1 -- 172.21.15.88:0/2870213217 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f35a4004eb0 con 0x7f35c006c250 2024-03-29T10:50:16.963 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:16.977+0000 7f35b6ffd640 10 client.?.objecter ms_handle_connect 0x7f35c00ae650 2024-03-29T10:50:16.963 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:16.977+0000 7f35b6ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:50:16.963 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:16.977+0000 7f35b6ffd640 1 -- 172.21.15.88:0/2870213217 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1585349588 0 0) 0x7f35a4004eb0 con 0x7f35c006c250 2024-03-29T10:50:16.963 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:16.977+0000 7f35b6ffd640 1 -- 172.21.15.88:0/2870213217 --> v1:172.21.15.88:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f35c0133b70 con 0x7f35c006c250 2024-03-29T10:50:16.963 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:16.977+0000 7f35b6ffd640 1 -- 172.21.15.88:0/2870213217 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f35b00018b0 con 0x7f35c006c250 2024-03-29T10:50:16.964 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:16.977+0000 7f35b6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:50:16.964 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:16.977+0000 7f35b6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:50:16.964 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:16.977+0000 7f35b6ffd640 10 monclient(hunting): dump: 2024-03-29T10:50:16.964 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:50:16.964 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:50:16.964 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:50:16.964 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:50:16.964 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:50:16.964 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:50:16.964 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:50:16.964 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:50:16.964 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:50:16.964 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:50:16.965 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:16.977+0000 7f35b6ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:50:16.965 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:16.978+0000 7f35b6ffd640 1 -- 172.21.15.88:0/2870213217 <== mon.2 v1:172.21.15.88:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1369814651 0 0) 0x7f35c0133b70 con 0x7f35c006c250 2024-03-29T10:50:16.965 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:16.978+0000 7f35b6ffd640 1 -- 172.21.15.88:0/2870213217 >> v1:172.21.15.196:6789/0 conn(0x7f35c013f8c0 legacy=0x7f35c0133460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:16.965 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:16.978+0000 7f35b6ffd640 1 -- 172.21.15.88:0/2870213217 >> v1:172.21.15.88:6789/0 conn(0x7f35c00ae650 legacy=0x7f35c00716b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:16.965 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:16.978+0000 7f35b6ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:50:16.965 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:16.978+0000 7f35b6ffd640 1 monclient: found mon.c 2024-03-29T10:50:16.965 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:16.978+0000 7f35b6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:50:16.965 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:16.978+0000 7f35b6ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-03-29T10:50:16.965 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:16.978+0000 7f35b6ffd640 1 -- 172.21.15.88:0/2870213217 --> v1:172.21.15.88:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35c0133d00 con 0x7f35c006c250 2024-03-29T10:50:16.965 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:16.978+0000 7f35b6ffd640 10 monclient: _finish_auth 0 2024-03-29T10:50:16.965 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:16.978+0000 7f35b6ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:50:16.965 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:16.978+0000 7f35b6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:50:16.965 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:16.978+0000 7f35c84c9640 5 monclient: authenticate success, global_id 4607 2024-03-29T10:50:16.965 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:16.978+0000 7f35c84c9640 10 monclient: _renew_subs 2024-03-29T10:50:16.965 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:16.978+0000 7f35c84c9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-03-29T10:50:16.966 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:16.978+0000 7f35c84c9640 1 -- 172.21.15.88:0/2870213217 --> v1:172.21.15.88:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35c0133f50 con 0x7f35c006c250 2024-03-29T10:50:16.966 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:16.978+0000 7f35c84c9640 10 client.4607.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:50:16.966 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:16.978+0000 7f35c84c9640 10 monclient: _renew_subs 2024-03-29T10:50:16.966 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:16.978+0000 7f35c84c9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-03-29T10:50:16.966 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:16.978+0000 7f35c84c9640 1 -- 172.21.15.88:0/2870213217 --> v1:172.21.15.88:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f35c01d7a70 con 0x7f35c006c250 2024-03-29T10:50:16.966 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:16.978+0000 7f35c84c9640 1 librados: init done 2024-03-29T10:50:16.966 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:16.979+0000 7f35b6ffd640 1 -- 172.21.15.88:0/2870213217 <== mon.2 v1:172.21.15.88:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f35b00047e0 con 0x7f35c006c250 2024-03-29T10:50:16.966 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:16.979+0000 7f35b6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:50:16.966 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:16.979+0000 7f35b6ffd640 1 -- 172.21.15.88:0/2870213217 <== mon.2 v1:172.21.15.88:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f35b0004af0 con 0x7f35c006c250 2024-03-29T10:50:16.966 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:16.979+0000 7f35b6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:50:16.966 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:16.979+0000 7f35b6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:50:16.966 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:16.979+0000 7f35b6ffd640 10 monclient: dump: 2024-03-29T10:50:16.966 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:50:16.966 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:50:16.967 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:50:16.967 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:50:16.967 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:50:16.967 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:50:16.967 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:50:16.967 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:50:16.967 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:50:16.967 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:50:16.967 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:16.980+0000 7f35c84c9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:50:16.967 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:16.980+0000 7f35c84c9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:50:16.967 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:16.980+0000 7f35c84c9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-03-29T10:50:16.967 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:16.980+0000 7f35c84c9640 1 -- 172.21.15.88:0/2870213217 --> v1:172.21.15.88:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35c006d480 con 0x7f35c006c250 2024-03-29T10:50:16.967 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:16.981+0000 7f35c4a62640 4 set_mon_vals no callback set 2024-03-29T10:50:16.968 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:16.981+0000 7f35b6ffd640 1 -- 172.21.15.88:0/2870213217 <== mon.2 v1:172.21.15.88:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4235629220 0 0) 0x7f35b001aac0 con 0x7f35c006c250 2024-03-29T10:50:16.968 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:16.983+0000 7f35b6ffd640 1 -- 172.21.15.88:0/2870213217 <== mon.2 v1:172.21.15.88:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4257673582 0 0) 0x7f35b00849b0 con 0x7f35c006c250 2024-03-29T10:50:16.968 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:16.983+0000 7f35b6ffd640 10 client.4607.objecter ms_dispatch 0x7f35c0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:50:16.968 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:16.983+0000 7f35b6ffd640 3 client.4607.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:50:16.968 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:16.983+0000 7f35b6ffd640 3 client.4607.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:50:16.969 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:16.983+0000 7f35b6ffd640 20 client.4607.objecter dump_active .. 0 homeless 2024-03-29T10:50:16.969 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:16.984+0000 7f35c5a64640 10 monclient: get_auth_request con 0x7f35a406dd60 auth_method 0 2024-03-29T10:50:16.969 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:16.984+0000 7f35b6ffd640 10 client.4607.objecter ms_handle_connect 0x7f35a406dd60 2024-03-29T10:50:16.974 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:16.993+0000 7f35b6ffd640 1 -- 172.21.15.88:0/2870213217 <== mon.2 v1:172.21.15.88:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f35b00500c0 con 0x7f35c006c250 2024-03-29T10:50:16.974 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:16.993+0000 7f35b6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:50:16.974 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:16.993+0000 7f35b6ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:50:17.297 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:17.315+0000 7f35c84c9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:50:17.298 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:17.315+0000 7f35c84c9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:50:17.298 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:17.315+0000 7f35c84c9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-03-29T10:50:17.298 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:17.315+0000 7f35c84c9640 1 -- 172.21.15.88:0/2870213217 --> v1:172.21.15.88:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f35c0071dc0 con 0x7f35c006c250 2024-03-29T10:50:17.298 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:17.316+0000 7f35b6ffd640 1 -- 172.21.15.88:0/2870213217 <== mon.2 v1:172.21.15.88:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 344506374) 0x7f35b00521c0 con 0x7f35c006c250 2024-03-29T10:50:17.298 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:17.316+0000 7f35b6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:50:17.298 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:17.316+0000 7f35b6ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:50:17.299 INFO:teuthology.misc.health.smithi088.stdout: 2024-03-29T10:50:17.299 INFO:teuthology.misc.health.smithi088.stdout:{"epoch":10,"fsid":"869f5188-f92b-42dd-a46d-8c646e0388a9","created":"2024-03-29T10:40:40.285087+0000","modified":"2024-03-29T10:40:42.536230+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:40:42.536230+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a26c0c1c-8de0-438c-a3f6-3df10ea61574","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5de24915-8dec-49b6-8b16-f3170197c591","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ac054af3-d589-4565-9644-e6af75975c53","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3ec4f045-c714-4a22-bd1a-e07330b45ab3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"cddb9867-8e30-4951-9b13-3d1e78b9c45a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dc6d9911-785d-432f-8b53-56a1ccab302f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"77c88437-b587-496e-8b4d-1824ea8b5305","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ce963d96-5a8e-44c3-99de-c0dbebe234ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:50:17.302 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:17.320+0000 7f35c84c9640 10 librados: watch_flush enter 2024-03-29T10:50:17.302 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:17.320+0000 7f35c84c9640 10 librados: watch_flush exit 2024-03-29T10:50:17.302 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:17.320+0000 7f35c84c9640 20 client.4607.objecter shutdown clearing up homeless session... 2024-03-29T10:50:17.302 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:17.320+0000 7f35c84c9640 10 client.4607.objecter successfully canceled tick 2024-03-29T10:50:17.302 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:17.320+0000 7f35c84c9640 1 -- 172.21.15.88:0/2870213217 >> v1:172.21.15.88:6816/1752861349 conn(0x7f35a406dd60 legacy=0x7f35a4070220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:17.302 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:17.320+0000 7f35c84c9640 10 monclient: shutdown 2024-03-29T10:50:17.302 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:17.320+0000 7f35c84c9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:50:17.302 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:17.320+0000 7f35c84c9640 1 -- 172.21.15.88:0/2870213217 >> v1:172.21.15.88:6790/0 conn(0x7f35c006c250 legacy=0x7f35c0141160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:17.303 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:17.320+0000 7f35c84c9640 1 -- 172.21.15.88:0/2870213217 shutdown_connections 2024-03-29T10:50:17.303 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:17.320+0000 7f35c84c9640 1 -- 172.21.15.88:0/2870213217 >> 172.21.15.88:0/2870213217 conn(0x7f35c0067610 msgr2=0x7f35c013ef80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:50:17.303 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:17.320+0000 7f35c84c9640 1 -- 172.21.15.88:0/2870213217 shutdown_connections 2024-03-29T10:50:17.303 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:17.320+0000 7f35c84c9640 1 -- 172.21.15.88:0/2870213217 wait complete. 2024-03-29T10:50:17.303 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:17.320+0000 7f35c84c9640 1 librados: shutdown 2024-03-29T10:50:17.317 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:50:24.123 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:50:24.275 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:24.293+0000 7f726c0d3640 10 monclient: get_monmap_and_config 2024-03-29T10:50:24.276 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:24.293+0000 7f726c0d3640 10 monclient: build_initial_monmap 2024-03-29T10:50:24.276 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:24.293+0000 7f726c0d3640 10 monclient: monmap: 2024-03-29T10:50:24.276 INFO:teuthology.misc.health.smithi088.stderr:epoch 0 2024-03-29T10:50:24.276 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:50:24.276 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:50:24.293448+0000 2024-03-29T10:50:24.277 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:50:24.293448+0000 2024-03-29T10:50:24.277 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 0 (unknown) 2024-03-29T10:50:24.277 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:50:24.277 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-03-29T10:50:24.277 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-29T10:50:24.277 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-b 2024-03-29T10:50:24.278 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:50:24.278 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:24.294+0000 7f726c0d3640 1 Processor -- start 2024-03-29T10:50:24.278 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:24.294+0000 7f726c0d3640 1 -- start start 2024-03-29T10:50:24.278 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:24.294+0000 7f726c0d3640 10 monclient: init 2024-03-29T10:50:24.294 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:24.295+0000 7f726c0d3640 10 monclient: _reopen_session rank -1 2024-03-29T10:50:24.294 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:24.295+0000 7f726c0d3640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:50:24.294 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:24.295+0000 7f726c0d3640 10 monclient(hunting): picked mon.noname-a con 0x7f72641422b0 addr v1:172.21.15.88:6789/0 2024-03-29T10:50:24.294 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:24.295+0000 7f726c0d3640 10 monclient(hunting): picked mon.noname-c con 0x7f7264142690 addr v1:172.21.15.196:6789/0 2024-03-29T10:50:24.294 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:24.295+0000 7f726c0d3640 10 monclient(hunting): picked mon.noname-b con 0x7f72640b17d0 addr v1:172.21.15.88:6790/0 2024-03-29T10:50:24.294 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:24.295+0000 7f726c0d3640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7264142c80 con 0x7f72641422b0 2024-03-29T10:50:24.294 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:24.295+0000 7f726c0d3640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72640aae40 con 0x7f7264142690 2024-03-29T10:50:24.294 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:24.295+0000 7f726c0d3640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f726413fbd0 con 0x7f72640b17d0 2024-03-29T10:50:24.295 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:24.295+0000 7f726c0d3640 10 monclient(hunting): _renew_subs 2024-03-29T10:50:24.295 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:24.295+0000 7f726c0d3640 10 monclient(hunting): authenticate will time out at 1346.530436s 2024-03-29T10:50:24.295 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:24.295+0000 7f726966e640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f7264142690 0x7f72640b2590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.88:49540/0 (socket says 172.21.15.88:49540) 2024-03-29T10:50:24.295 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:24.295+0000 7f726966e640 1 -- 172.21.15.88:0/1268296464 learned_addr learned my addr 172.21.15.88:0/1268296464 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-29T10:50:24.295 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:24.296+0000 7f7268e6d640 1 -- 172.21.15.88:0/1268296464 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1956630728 0 0) 0x7f72640aae40 con 0x7f7264142690 2024-03-29T10:50:24.295 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:24.296+0000 7f7268e6d640 10 monclient(hunting): _negotiate 2024-03-29T10:50:24.295 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:24.296+0000 7f7268e6d640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:50:24.295 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:24.296+0000 7f7268e6d640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:50:24.295 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:24.296+0000 7f7268e6d640 10 monclient(hunting): my global_id is 4558 2024-03-29T10:50:24.295 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:24.296+0000 7f7268e6d640 1 -- 172.21.15.88:0/1268296464 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7250003520 con 0x7f7264142690 2024-03-29T10:50:24.295 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:24.296+0000 7f7268e6d640 1 -- 172.21.15.88:0/1268296464 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2817788362 0 0) 0x7f726413fbd0 con 0x7f72640b17d0 2024-03-29T10:50:24.295 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:24.296+0000 7f7268e6d640 10 monclient(hunting): _negotiate 2024-03-29T10:50:24.296 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:24.296+0000 7f7268e6d640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:50:24.296 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:24.296+0000 7f7268e6d640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:50:24.296 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:24.296+0000 7f7268e6d640 10 monclient(hunting): my global_id is 4610 2024-03-29T10:50:24.296 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:24.296+0000 7f7268e6d640 1 -- 172.21.15.88:0/1268296464 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f72640aae40 con 0x7f72640b17d0 2024-03-29T10:50:24.296 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:24.296+0000 7f7268e6d640 1 -- 172.21.15.88:0/1268296464 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 566654526 0 0) 0x7f7250003520 con 0x7f7264142690 2024-03-29T10:50:24.296 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:24.296+0000 7f7268e6d640 1 -- 172.21.15.88:0/1268296464 >> v1:172.21.15.88:6790/0 conn(0x7f72640b17d0 legacy=0x7f72640b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:24.296 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:24.296+0000 7f7268e6d640 1 -- 172.21.15.88:0/1268296464 >> v1:172.21.15.88:6789/0 conn(0x7f72641422b0 legacy=0x7f726413f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:24.296 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:24.296+0000 7f7268e6d640 10 monclient: _finish_hunting 0 2024-03-29T10:50:24.296 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:24.296+0000 7f7268e6d640 1 monclient: found mon.noname-c 2024-03-29T10:50:24.296 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:24.296+0000 7f7268e6d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:50:24.296 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:24.296+0000 7f7268e6d640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.196:6789/0 2024-03-29T10:50:24.296 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:24.296+0000 7f7268e6d640 1 -- 172.21.15.88:0/1268296464 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7264151940 con 0x7f7264142690 2024-03-29T10:50:24.296 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:24.296+0000 7f7268e6d640 10 monclient: _finish_auth 0 2024-03-29T10:50:24.297 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:24.296+0000 7f7268e6d640 10 monclient: _check_auth_tickets 2024-03-29T10:50:24.297 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:24.296+0000 7f7268e6d640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:50:24.297 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:24.296+0000 7f726c0d3640 5 monclient: authenticate success, global_id 4558 2024-03-29T10:50:24.297 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:24.296+0000 7f726c0d3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:50:24.297 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:24.296+0000 7f7268e6d640 1 -- 172.21.15.88:0/1268296464 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f7254001690 con 0x7f7264142690 2024-03-29T10:50:24.297 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:24.296+0000 7f7268e6d640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:50:24.297 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:24.296+0000 7f7268e6d640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:50:24.297 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:24.296+0000 7f7268e6d640 10 monclient: dump: 2024-03-29T10:50:24.297 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:50:24.297 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:50:24.297 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:50:24.298 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:50:24.298 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:50:24.298 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:50:24.298 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:50:24.298 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:50:24.298 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:50:24.298 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:50:24.298 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:24.296+0000 7f726c0d3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:50:24.298 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:24.297+0000 7f7268e6d640 1 -- 172.21.15.88:0/1268296464 <== mon.1 v1:172.21.15.196:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f72540017f0 con 0x7f7264142690 2024-03-29T10:50:24.298 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:24.297+0000 7f7268e6d640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:50:24.298 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:24.297+0000 7f726c0d3640 10 monclient: get_monmap_and_config success 2024-03-29T10:50:24.298 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:24.297+0000 7f726c0d3640 4 set_mon_vals no callback set 2024-03-29T10:50:24.299 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:24.297+0000 7f7268e6d640 1 -- 172.21.15.88:0/1268296464 <== mon.1 v1:172.21.15.196:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f7254001b80 con 0x7f7264142690 2024-03-29T10:50:24.299 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:24.297+0000 7f7268e6d640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:50:24.299 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:24.297+0000 7f7268e6d640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:50:24.299 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:24.297+0000 7f726c0d3640 10 monclient: shutdown 2024-03-29T10:50:24.299 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:24.298+0000 7f7268e6d640 10 monclient: dump: 2024-03-29T10:50:24.299 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:50:24.299 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:50:24.299 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:50:24.299 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:50:24.299 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:50:24.299 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:50:24.299 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:50:24.300 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:50:24.300 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:50:24.300 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:50:24.300 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:24.298+0000 7f726c0d3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:50:24.300 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:24.298+0000 7f726c0d3640 1 -- 172.21.15.88:0/1268296464 >> v1:172.21.15.196:6789/0 conn(0x7f7264142690 legacy=0x7f72640b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:24.300 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:24.298+0000 7f726c0d3640 1 -- 172.21.15.88:0/1268296464 shutdown_connections 2024-03-29T10:50:24.300 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:24.298+0000 7f726c0d3640 1 -- 172.21.15.88:0/1268296464 >> 172.21.15.88:0/1268296464 conn(0x7f7264066a00 msgr2=0x7f7264066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:50:24.300 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:24.298+0000 7f726c0d3640 1 -- 172.21.15.88:0/1268296464 shutdown_connections 2024-03-29T10:50:24.300 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:24.298+0000 7f726c0d3640 1 -- 172.21.15.88:0/1268296464 wait complete. 2024-03-29T10:50:24.300 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:24.299+0000 7f726c0d3640 10 monclient: build_initial_monmap 2024-03-29T10:50:24.300 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:24.299+0000 7f726c0d3640 10 monclient: monmap: 2024-03-29T10:50:24.300 INFO:teuthology.misc.health.smithi088.stderr:epoch 0 2024-03-29T10:50:24.301 INFO:teuthology.misc.health.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:50:24.301 INFO:teuthology.misc.health.smithi088.stderr:last_changed 0.000000 2024-03-29T10:50:24.301 INFO:teuthology.misc.health.smithi088.stderr:created 0.000000 2024-03-29T10:50:24.301 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 0 (unknown) 2024-03-29T10:50:24.301 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:50:24.301 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-03-29T10:50:24.301 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-29T10:50:24.301 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-c 2024-03-29T10:50:24.301 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:50:24.301 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:24.299+0000 7f726c0d3640 1 librados: starting msgr at 2024-03-29T10:50:24.301 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:24.300+0000 7f726c0d3640 1 librados: starting objecter 2024-03-29T10:50:24.301 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:24.300+0000 7f726c0d3640 1 Processor -- start 2024-03-29T10:50:24.301 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:24.300+0000 7f726c0d3640 1 -- start start 2024-03-29T10:50:24.302 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:24.300+0000 7f726c0d3640 1 librados: setting wanted keys 2024-03-29T10:50:24.302 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:24.300+0000 7f726c0d3640 1 librados: calling monclient init 2024-03-29T10:50:24.302 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:24.300+0000 7f726c0d3640 10 monclient: init 2024-03-29T10:50:24.302 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:24.301+0000 7f726c0d3640 10 monclient: _reopen_session rank -1 2024-03-29T10:50:24.302 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:24.301+0000 7f726c0d3640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:50:24.302 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:24.301+0000 7f726c0d3640 10 monclient(hunting): picked mon.noname-b con 0x7f72640b17d0 addr v1:172.21.15.196:6789/0 2024-03-29T10:50:24.302 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:24.301+0000 7f7269e6f640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f72640b17d0 0x7f7264071920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.88:50266/0 (socket says 172.21.15.88:50266) 2024-03-29T10:50:24.302 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:24.301+0000 7f7269e6f640 1 -- 172.21.15.88:0/1591428951 learned_addr learned my addr 172.21.15.88:0/1591428951 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-29T10:50:24.302 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:24.301+0000 7f726c0d3640 10 monclient(hunting): picked mon.noname-a con 0x7f72641422b0 addr v1:172.21.15.88:6789/0 2024-03-29T10:50:24.302 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:24.302+0000 7f726c0d3640 10 monclient(hunting): picked mon.noname-c con 0x7f7264142690 addr v1:172.21.15.88:6790/0 2024-03-29T10:50:24.303 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:24.302+0000 7f726c0d3640 1 -- 172.21.15.88:0/1591428951 --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f726414dd70 con 0x7f72641422b0 2024-03-29T10:50:24.303 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:24.302+0000 7f726c0d3640 1 -- 172.21.15.88:0/1591428951 --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7264072030 con 0x7f72640b17d0 2024-03-29T10:50:24.303 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:24.302+0000 7f726c0d3640 1 -- 172.21.15.88:0/1591428951 --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7264071440 con 0x7f7264142690 2024-03-29T10:50:24.303 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:24.302+0000 7f726c0d3640 10 monclient(hunting): _renew_subs 2024-03-29T10:50:24.303 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:24.302+0000 7f726c0d3640 10 monclient(hunting): authenticate will time out at 1346.537236s 2024-03-29T10:50:24.303 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:24.302+0000 7f725a7fc640 10 client.?.objecter ms_handle_connect 0x7f72640b17d0 2024-03-29T10:50:24.303 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:24.302+0000 7f725a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:50:24.303 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:24.302+0000 7f725a7fc640 10 client.?.objecter ms_handle_connect 0x7f72641422b0 2024-03-29T10:50:24.304 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:24.302+0000 7f725a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:50:24.304 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:24.302+0000 7f725a7fc640 10 client.?.objecter ms_handle_connect 0x7f7264142690 2024-03-29T10:50:24.304 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:24.302+0000 7f725a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:50:24.304 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:24.302+0000 7f725a7fc640 1 -- 172.21.15.88:0/1591428951 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2233149968 0 0) 0x7f726414dd70 con 0x7f72641422b0 2024-03-29T10:50:24.304 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:24.302+0000 7f725a7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:50:24.304 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:24.302+0000 7f725a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:50:24.304 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:24.302+0000 7f725a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:50:24.304 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:24.302+0000 7f725a7fc640 10 monclient(hunting): my global_id is 4578 2024-03-29T10:50:24.304 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:24.302+0000 7f725a7fc640 1 -- 172.21.15.88:0/1591428951 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7248004eb0 con 0x7f72641422b0 2024-03-29T10:50:24.304 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:24.302+0000 7f725a7fc640 1 -- 172.21.15.88:0/1591428951 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2380292958 0 0) 0x7f7264072030 con 0x7f72640b17d0 2024-03-29T10:50:24.304 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:24.302+0000 7f725a7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:50:24.304 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:24.302+0000 7f725a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:50:24.305 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:24.302+0000 7f725a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:50:24.305 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:24.302+0000 7f725a7fc640 10 monclient(hunting): my global_id is 4561 2024-03-29T10:50:24.305 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:24.302+0000 7f725a7fc640 1 -- 172.21.15.88:0/1591428951 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f726414dd70 con 0x7f72640b17d0 2024-03-29T10:50:24.305 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:24.303+0000 7f725a7fc640 1 -- 172.21.15.88:0/1591428951 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4179502291 0 0) 0x7f7264071440 con 0x7f7264142690 2024-03-29T10:50:24.305 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:24.303+0000 7f725a7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:50:24.305 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:24.303+0000 7f725a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:50:24.305 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:24.303+0000 7f725a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:50:24.305 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:24.303+0000 7f725a7fc640 10 monclient(hunting): my global_id is 4613 2024-03-29T10:50:24.305 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:24.303+0000 7f725a7fc640 1 -- 172.21.15.88:0/1591428951 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7264072030 con 0x7f7264142690 2024-03-29T10:50:24.306 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:24.303+0000 7f725a7fc640 1 -- 172.21.15.88:0/1591428951 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2814505492 0 0) 0x7f7248004eb0 con 0x7f72641422b0 2024-03-29T10:50:24.306 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:24.303+0000 7f725a7fc640 1 -- 172.21.15.88:0/1591428951 --> v1:172.21.15.88:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7264071440 con 0x7f72641422b0 2024-03-29T10:50:24.306 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:24.303+0000 7f725a7fc640 1 -- 172.21.15.88:0/1591428951 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1438055346 0 0) 0x7f726414dd70 con 0x7f72640b17d0 2024-03-29T10:50:24.306 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:24.303+0000 7f725a7fc640 1 -- 172.21.15.88:0/1591428951 --> v1:172.21.15.196:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7248004eb0 con 0x7f72640b17d0 2024-03-29T10:50:24.306 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:24.303+0000 7f725a7fc640 1 -- 172.21.15.88:0/1591428951 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f7254001b00 con 0x7f72641422b0 2024-03-29T10:50:24.306 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:24.303+0000 7f725a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:50:24.306 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:24.303+0000 7f725a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:50:24.306 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:24.303+0000 7f725a7fc640 10 monclient(hunting): dump: 2024-03-29T10:50:24.307 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:50:24.307 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:50:24.307 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:50:24.307 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:50:24.307 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:50:24.307 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:50:24.307 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:50:24.308 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:50:24.308 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:50:24.308 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:50:24.308 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:24.303+0000 7f725a7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:50:24.308 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:24.303+0000 7f725a7fc640 1 -- 172.21.15.88:0/1591428951 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f7260001930 con 0x7f72640b17d0 2024-03-29T10:50:24.308 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:24.303+0000 7f725a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:50:24.308 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:24.303+0000 7f725a7fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T10:50:24.308 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:24.303+0000 7f725a7fc640 10 monclient(hunting): dump: 2024-03-29T10:50:24.308 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:50:24.309 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:50:24.309 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:50:24.309 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:50:24.309 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:50:24.309 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:50:24.310 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:50:24.310 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:50:24.310 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:50:24.310 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:50:24.310 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:24.303+0000 7f725a7fc640 1 -- 172.21.15.88:0/1591428951 <== mon.0 v1:172.21.15.88:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3609750524 0 0) 0x7f7264071440 con 0x7f72641422b0 2024-03-29T10:50:24.310 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:24.303+0000 7f725a7fc640 1 -- 172.21.15.88:0/1591428951 >> v1:172.21.15.88:6790/0 conn(0x7f7264142690 legacy=0x7f72641eeb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:24.311 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:24.303+0000 7f725a7fc640 1 -- 172.21.15.88:0/1591428951 >> v1:172.21.15.196:6789/0 conn(0x7f72640b17d0 legacy=0x7f7264071920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:24.311 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:24.303+0000 7f725a7fc640 10 monclient: _finish_hunting 0 2024-03-29T10:50:24.311 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:24.303+0000 7f725a7fc640 1 monclient: found mon.a 2024-03-29T10:50:24.311 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:24.304+0000 7f725a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:50:24.311 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:24.304+0000 7f725a7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-03-29T10:50:24.311 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:24.304+0000 7f725a7fc640 1 -- 172.21.15.88:0/1591428951 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72640ac230 con 0x7f72641422b0 2024-03-29T10:50:24.311 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:24.304+0000 7f725a7fc640 10 monclient: _finish_auth 0 2024-03-29T10:50:24.311 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:24.304+0000 7f725a7fc640 10 monclient: _check_auth_tickets 2024-03-29T10:50:24.311 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:24.304+0000 7f725a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:50:24.311 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:24.304+0000 7f726c0d3640 5 monclient: authenticate success, global_id 4578 2024-03-29T10:50:24.311 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:24.304+0000 7f726c0d3640 10 monclient: _renew_subs 2024-03-29T10:50:24.311 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:24.304+0000 7f726c0d3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-03-29T10:50:24.312 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:24.304+0000 7f726c0d3640 1 -- 172.21.15.88:0/1591428951 --> v1:172.21.15.88:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72640ac510 con 0x7f72641422b0 2024-03-29T10:50:24.312 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:24.304+0000 7f726c0d3640 10 client.4578.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:50:24.312 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:24.304+0000 7f726c0d3640 10 monclient: _renew_subs 2024-03-29T10:50:24.312 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:24.304+0000 7f726c0d3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-03-29T10:50:24.312 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:24.304+0000 7f726c0d3640 1 -- 172.21.15.88:0/1591428951 --> v1:172.21.15.88:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f72640ad040 con 0x7f72641422b0 2024-03-29T10:50:24.312 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:24.304+0000 7f726c0d3640 1 librados: init done 2024-03-29T10:50:24.312 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:24.305+0000 7f725a7fc640 1 -- 172.21.15.88:0/1591428951 <== mon.0 v1:172.21.15.88:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7254001220 con 0x7f72641422b0 2024-03-29T10:50:24.312 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:24.305+0000 7f725a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:50:24.312 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:24.305+0000 7f725a7fc640 1 -- 172.21.15.88:0/1591428951 <== mon.0 v1:172.21.15.88:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f7254004a90 con 0x7f72641422b0 2024-03-29T10:50:24.312 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:24.305+0000 7f725a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:50:24.312 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:24.305+0000 7f725a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:50:24.312 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:24.305+0000 7f725a7fc640 10 monclient: dump: 2024-03-29T10:50:24.312 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:50:24.312 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:50:24.313 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:50:24.313 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:50:24.313 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:50:24.313 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:50:24.313 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:50:24.313 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:50:24.313 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:50:24.313 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:50:24.313 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:24.305+0000 7f725bfff640 4 set_mon_vals no callback set 2024-03-29T10:50:24.313 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:24.305+0000 7f726c0d3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:50:24.313 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:24.306+0000 7f726c0d3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:50:24.313 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:24.306+0000 7f726c0d3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-03-29T10:50:24.313 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:24.306+0000 7f726c0d3640 1 -- 172.21.15.88:0/1591428951 --> v1:172.21.15.88:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f724c005080 con 0x7f72641422b0 2024-03-29T10:50:24.313 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:24.307+0000 7f725a7fc640 1 -- 172.21.15.88:0/1591428951 <== mon.0 v1:172.21.15.88:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4235629220 0 0) 0x7f725401aa60 con 0x7f72641422b0 2024-03-29T10:50:24.314 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:24.308+0000 7f7269e6f640 10 monclient: get_auth_request con 0x7f724806e040 auth_method 0 2024-03-29T10:50:24.314 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:24.309+0000 7f725a7fc640 10 client.4578.objecter ms_handle_connect 0x7f724806e040 2024-03-29T10:50:24.314 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:24.309+0000 7f725a7fc640 1 -- 172.21.15.88:0/1591428951 <== mon.0 v1:172.21.15.88:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4257673582 0 0) 0x7f7254084960 con 0x7f72641422b0 2024-03-29T10:50:24.314 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:24.309+0000 7f725a7fc640 10 client.4578.objecter ms_dispatch 0x7f7264000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:50:24.314 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:24.309+0000 7f725a7fc640 3 client.4578.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:50:24.314 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:24.309+0000 7f725a7fc640 3 client.4578.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:50:24.314 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:24.309+0000 7f725a7fc640 20 client.4578.objecter dump_active .. 0 homeless 2024-03-29T10:50:24.315 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:24.318+0000 7f725a7fc640 1 -- 172.21.15.88:0/1591428951 <== mon.0 v1:172.21.15.88:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7254050240 con 0x7f72641422b0 2024-03-29T10:50:24.315 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:24.318+0000 7f725a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:50:24.315 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:24.318+0000 7f725a7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:50:24.673 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:24.691+0000 7f726c0d3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:50:24.673 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:24.691+0000 7f726c0d3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:50:24.673 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:24.691+0000 7f726c0d3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-03-29T10:50:24.673 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:24.691+0000 7f726c0d3640 1 -- 172.21.15.88:0/1591428951 --> v1:172.21.15.88:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f724c005330 con 0x7f72641422b0 2024-03-29T10:50:24.674 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:24.692+0000 7f725a7fc640 1 -- 172.21.15.88:0/1591428951 <== mon.0 v1:172.21.15.88:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 344506374) 0x7f7254052170 con 0x7f72641422b0 2024-03-29T10:50:24.674 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:24.692+0000 7f725a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:50:24.674 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:24.692+0000 7f725a7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:50:24.674 INFO:teuthology.misc.health.smithi088.stdout: 2024-03-29T10:50:24.674 INFO:teuthology.misc.health.smithi088.stdout:{"epoch":10,"fsid":"869f5188-f92b-42dd-a46d-8c646e0388a9","created":"2024-03-29T10:40:40.285087+0000","modified":"2024-03-29T10:40:42.536230+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:40:42.536230+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a26c0c1c-8de0-438c-a3f6-3df10ea61574","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5de24915-8dec-49b6-8b16-f3170197c591","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ac054af3-d589-4565-9644-e6af75975c53","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3ec4f045-c714-4a22-bd1a-e07330b45ab3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"cddb9867-8e30-4951-9b13-3d1e78b9c45a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dc6d9911-785d-432f-8b53-56a1ccab302f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"77c88437-b587-496e-8b4d-1824ea8b5305","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ce963d96-5a8e-44c3-99de-c0dbebe234ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:50:24.679 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:24.695+0000 7f726c0d3640 10 librados: watch_flush enter 2024-03-29T10:50:24.679 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:24.695+0000 7f726c0d3640 10 librados: watch_flush exit 2024-03-29T10:50:24.679 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:24.695+0000 7f726c0d3640 20 client.4578.objecter shutdown clearing up homeless session... 2024-03-29T10:50:24.679 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:24.695+0000 7f726c0d3640 10 client.4578.objecter successfully canceled tick 2024-03-29T10:50:24.679 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:24.696+0000 7f726c0d3640 1 -- 172.21.15.88:0/1591428951 >> v1:172.21.15.88:6816/1752861349 conn(0x7f724806e040 legacy=0x7f7248070500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:24.679 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:24.696+0000 7f726c0d3640 10 monclient: shutdown 2024-03-29T10:50:24.679 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:24.696+0000 7f726c0d3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:50:24.679 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:24.696+0000 7f726c0d3640 1 -- 172.21.15.88:0/1591428951 >> v1:172.21.15.88:6789/0 conn(0x7f72641422b0 legacy=0x7f72640ac930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:24.679 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:24.696+0000 7f726c0d3640 1 -- 172.21.15.88:0/1591428951 shutdown_connections 2024-03-29T10:50:24.679 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:24.696+0000 7f726c0d3640 1 -- 172.21.15.88:0/1591428951 >> 172.21.15.88:0/1591428951 conn(0x7f7264066a00 msgr2=0x7f7264066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:50:24.679 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:24.696+0000 7f726c0d3640 1 -- 172.21.15.88:0/1591428951 shutdown_connections 2024-03-29T10:50:24.679 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:24.696+0000 7f726c0d3640 1 -- 172.21.15.88:0/1591428951 wait complete. 2024-03-29T10:50:24.680 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:24.696+0000 7f726c0d3640 1 librados: shutdown 2024-03-29T10:50:24.693 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:50:31.500 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:50:31.644 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:31.663+0000 7f09caf9d640 10 monclient: get_monmap_and_config 2024-03-29T10:50:31.645 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:31.663+0000 7f09caf9d640 10 monclient: build_initial_monmap 2024-03-29T10:50:31.645 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:31.663+0000 7f09caf9d640 10 monclient: monmap: 2024-03-29T10:50:31.645 INFO:teuthology.misc.health.smithi088.stderr:epoch 0 2024-03-29T10:50:31.645 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:50:31.645 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:50:31.664204+0000 2024-03-29T10:50:31.645 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:50:31.664204+0000 2024-03-29T10:50:31.645 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 0 (unknown) 2024-03-29T10:50:31.645 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:50:31.645 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-03-29T10:50:31.645 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-29T10:50:31.645 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-b 2024-03-29T10:50:31.645 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:50:31.646 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:31.664+0000 7f09caf9d640 1 Processor -- start 2024-03-29T10:50:31.646 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:31.664+0000 7f09caf9d640 1 -- start start 2024-03-29T10:50:31.646 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:31.664+0000 7f09caf9d640 10 monclient: init 2024-03-29T10:50:31.647 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:31.665+0000 7f09caf9d640 10 monclient: _reopen_session rank -1 2024-03-29T10:50:31.647 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:31.665+0000 7f09caf9d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:50:31.647 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:31.665+0000 7f09caf9d640 10 monclient(hunting): picked mon.noname-b con 0x7f09c41422b0 addr v1:172.21.15.88:6790/0 2024-03-29T10:50:31.647 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:31.665+0000 7f09caf9d640 10 monclient(hunting): picked mon.noname-c con 0x7f09c4142690 addr v1:172.21.15.196:6789/0 2024-03-29T10:50:31.647 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:31.665+0000 7f09caf9d640 10 monclient(hunting): picked mon.noname-a con 0x7f09c40b17d0 addr v1:172.21.15.88:6789/0 2024-03-29T10:50:31.647 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:31.665+0000 7f09caf9d640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09c4142c80 con 0x7f09c40b17d0 2024-03-29T10:50:31.647 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:31.665+0000 7f09caf9d640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09c40aae40 con 0x7f09c4142690 2024-03-29T10:50:31.647 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:31.665+0000 7f09caf9d640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09c413fbd0 con 0x7f09c41422b0 2024-03-29T10:50:31.647 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:31.665+0000 7f09caf9d640 10 monclient(hunting): _renew_subs 2024-03-29T10:50:31.647 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:31.665+0000 7f09caf9d640 10 monclient(hunting): authenticate will time out at 1353.901472s 2024-03-29T10:50:31.648 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:31.665+0000 7f09c8d39640 1 --1- >> v1:172.21.15.88:6790/0 conn(0x7f09c41422b0 0x7f09c413f4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6790/0 says I am v1:172.21.15.88:48682/0 (socket says 172.21.15.88:48682) 2024-03-29T10:50:31.648 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:31.665+0000 7f09c8d39640 1 -- 172.21.15.88:0/2315133867 learned_addr learned my addr 172.21.15.88:0/2315133867 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-29T10:50:31.651 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:31.666+0000 7f09c37fe640 1 -- 172.21.15.88:0/2315133867 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3182782399 0 0) 0x7f09c40aae40 con 0x7f09c4142690 2024-03-29T10:50:31.652 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:31.666+0000 7f09c37fe640 10 monclient(hunting): _negotiate 2024-03-29T10:50:31.652 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:31.666+0000 7f09c37fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:50:31.652 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:31.666+0000 7f09c37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:50:31.652 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:31.666+0000 7f09c37fe640 10 monclient(hunting): my global_id is 4564 2024-03-29T10:50:31.652 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:31.666+0000 7f09c37fe640 1 -- 172.21.15.88:0/2315133867 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f09b0003520 con 0x7f09c4142690 2024-03-29T10:50:31.652 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:31.667+0000 7f09c37fe640 1 -- 172.21.15.88:0/2315133867 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 179436257 0 0) 0x7f09c413fbd0 con 0x7f09c41422b0 2024-03-29T10:50:31.652 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:31.667+0000 7f09c37fe640 10 monclient(hunting): _negotiate 2024-03-29T10:50:31.652 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:31.667+0000 7f09c37fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:50:31.652 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:31.667+0000 7f09c37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:50:31.652 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:31.667+0000 7f09c37fe640 10 monclient(hunting): my global_id is 4616 2024-03-29T10:50:31.652 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:31.667+0000 7f09c37fe640 1 -- 172.21.15.88:0/2315133867 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f09c40aae40 con 0x7f09c41422b0 2024-03-29T10:50:31.652 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:31.667+0000 7f09c37fe640 1 -- 172.21.15.88:0/2315133867 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3332727629 0 0) 0x7f09c40aae40 con 0x7f09c41422b0 2024-03-29T10:50:31.652 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:31.667+0000 7f09c37fe640 1 -- 172.21.15.88:0/2315133867 >> v1:172.21.15.196:6789/0 conn(0x7f09c4142690 legacy=0x7f09c40b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:31.652 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:31.667+0000 7f09c37fe640 1 -- 172.21.15.88:0/2315133867 >> v1:172.21.15.88:6789/0 conn(0x7f09c40b17d0 legacy=0x7f09c40b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:31.653 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:31.667+0000 7f09c37fe640 10 monclient: _finish_hunting 0 2024-03-29T10:50:31.653 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:31.667+0000 7f09c37fe640 1 monclient: found mon.noname-b 2024-03-29T10:50:31.653 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:31.667+0000 7f09c37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:50:31.653 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:31.667+0000 7f09c37fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.88:6790/0 2024-03-29T10:50:31.653 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:31.667+0000 7f09c37fe640 1 -- 172.21.15.88:0/2315133867 --> v1:172.21.15.88:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09c4151940 con 0x7f09c41422b0 2024-03-29T10:50:31.653 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:31.667+0000 7f09c37fe640 10 monclient: _finish_auth 0 2024-03-29T10:50:31.653 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:31.667+0000 7f09c37fe640 10 monclient: _check_auth_tickets 2024-03-29T10:50:31.653 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:31.667+0000 7f09c37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:50:31.653 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:31.667+0000 7f09caf9d640 5 monclient: authenticate success, global_id 4616 2024-03-29T10:50:31.653 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:31.667+0000 7f09caf9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:50:31.653 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:31.667+0000 7f09c37fe640 1 -- 172.21.15.88:0/2315133867 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f09bc0019f0 con 0x7f09c41422b0 2024-03-29T10:50:31.653 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:31.667+0000 7f09c37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:50:31.653 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:31.667+0000 7f09c37fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:50:31.653 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:31.667+0000 7f09c37fe640 10 monclient: dump: 2024-03-29T10:50:31.653 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:50:31.654 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:50:31.654 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:50:31.654 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:50:31.654 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:50:31.654 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:50:31.654 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:50:31.654 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:50:31.654 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:50:31.654 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:50:31.654 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:31.667+0000 7f09caf9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:50:31.655 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:31.668+0000 7f09c37fe640 1 -- 172.21.15.88:0/2315133867 <== mon.2 v1:172.21.15.88:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f09bc001bd0 con 0x7f09c41422b0 2024-03-29T10:50:31.655 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:31.668+0000 7f09c37fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:50:31.655 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:31.668+0000 7f09caf9d640 10 monclient: get_monmap_and_config success 2024-03-29T10:50:31.655 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:31.668+0000 7f09caf9d640 4 set_mon_vals no callback set 2024-03-29T10:50:31.655 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:31.668+0000 7f09c37fe640 1 -- 172.21.15.88:0/2315133867 <== mon.2 v1:172.21.15.88:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f09bc001f60 con 0x7f09c41422b0 2024-03-29T10:50:31.655 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:31.668+0000 7f09c37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:50:31.655 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:31.668+0000 7f09c37fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:50:31.655 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:31.668+0000 7f09c37fe640 10 monclient: dump: 2024-03-29T10:50:31.655 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:50:31.655 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:50:31.655 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:50:31.655 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:50:31.655 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:50:31.655 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:50:31.655 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:50:31.656 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:50:31.656 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:50:31.656 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:50:31.656 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:31.668+0000 7f09caf9d640 10 monclient: shutdown 2024-03-29T10:50:31.656 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:31.668+0000 7f09caf9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:50:31.656 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:31.668+0000 7f09caf9d640 1 -- 172.21.15.88:0/2315133867 >> v1:172.21.15.88:6790/0 conn(0x7f09c41422b0 legacy=0x7f09c413f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:31.656 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:31.669+0000 7f09caf9d640 1 -- 172.21.15.88:0/2315133867 shutdown_connections 2024-03-29T10:50:31.656 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:31.669+0000 7f09caf9d640 1 -- 172.21.15.88:0/2315133867 >> 172.21.15.88:0/2315133867 conn(0x7f09c4066a00 msgr2=0x7f09c4066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:50:31.656 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:31.669+0000 7f09caf9d640 1 -- 172.21.15.88:0/2315133867 shutdown_connections 2024-03-29T10:50:31.656 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:31.669+0000 7f09caf9d640 1 -- 172.21.15.88:0/2315133867 wait complete. 2024-03-29T10:50:31.656 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:31.669+0000 7f09caf9d640 10 monclient: build_initial_monmap 2024-03-29T10:50:31.656 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:31.669+0000 7f09caf9d640 10 monclient: monmap: 2024-03-29T10:50:31.656 INFO:teuthology.misc.health.smithi088.stderr:epoch 0 2024-03-29T10:50:31.656 INFO:teuthology.misc.health.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:50:31.656 INFO:teuthology.misc.health.smithi088.stderr:last_changed 0.000000 2024-03-29T10:50:31.657 INFO:teuthology.misc.health.smithi088.stderr:created 0.000000 2024-03-29T10:50:31.657 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 0 (unknown) 2024-03-29T10:50:31.657 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:50:31.657 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-03-29T10:50:31.657 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-29T10:50:31.657 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-c 2024-03-29T10:50:31.657 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:50:31.664 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:31.670+0000 7f09caf9d640 1 librados: starting msgr at 2024-03-29T10:50:31.665 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:31.670+0000 7f09caf9d640 1 librados: starting objecter 2024-03-29T10:50:31.665 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:31.670+0000 7f09caf9d640 1 Processor -- start 2024-03-29T10:50:31.665 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:31.670+0000 7f09caf9d640 1 -- start start 2024-03-29T10:50:31.665 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:31.670+0000 7f09caf9d640 1 librados: setting wanted keys 2024-03-29T10:50:31.665 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:31.670+0000 7f09caf9d640 1 librados: calling monclient init 2024-03-29T10:50:31.665 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:31.670+0000 7f09caf9d640 10 monclient: init 2024-03-29T10:50:31.665 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:31.671+0000 7f09caf9d640 10 monclient: _reopen_session rank -1 2024-03-29T10:50:31.665 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:31.671+0000 7f09caf9d640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:50:31.665 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:31.671+0000 7f09caf9d640 10 monclient(hunting): picked mon.noname-c con 0x7f09c40b17d0 addr v1:172.21.15.88:6790/0 2024-03-29T10:50:31.665 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:31.671+0000 7f09caf9d640 10 monclient(hunting): picked mon.noname-a con 0x7f09c41422b0 addr v1:172.21.15.88:6789/0 2024-03-29T10:50:31.665 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:31.671+0000 7f09c8d39640 1 --1- >> v1:172.21.15.88:6790/0 conn(0x7f09c40b17d0 0x7f09c414f280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6790/0 says I am v1:172.21.15.88:49046/0 (socket says 172.21.15.88:49046) 2024-03-29T10:50:31.665 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:31.671+0000 7f09caf9d640 10 monclient(hunting): picked mon.noname-b con 0x7f09c4142690 addr v1:172.21.15.196:6789/0 2024-03-29T10:50:31.665 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:31.671+0000 7f09caf9d640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09c406f080 con 0x7f09c41422b0 2024-03-29T10:50:31.666 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:31.672+0000 7f09caf9d640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09c406f360 con 0x7f09c4142690 2024-03-29T10:50:31.666 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:31.672+0000 7f09caf9d640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09c406de90 con 0x7f09c40b17d0 2024-03-29T10:50:31.666 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:31.672+0000 7f09caf9d640 10 monclient(hunting): _renew_subs 2024-03-29T10:50:31.666 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:31.672+0000 7f09caf9d640 10 monclient(hunting): authenticate will time out at 1353.907967s 2024-03-29T10:50:31.666 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:31.672+0000 7f09c8d39640 1 -- 172.21.15.88:0/450823292 learned_addr learned my addr 172.21.15.88:0/450823292 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-29T10:50:31.666 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:31.672+0000 7f09c17fa640 10 client.?.objecter ms_handle_connect 0x7f09c40b17d0 2024-03-29T10:50:31.666 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:31.672+0000 7f09c17fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:50:31.666 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:31.672+0000 7f09c17fa640 10 client.?.objecter ms_handle_connect 0x7f09c41422b0 2024-03-29T10:50:31.666 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:31.672+0000 7f09c17fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:50:31.666 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:31.672+0000 7f09c17fa640 1 -- 172.21.15.88:0/450823292 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2076028758 0 0) 0x7f09c406de90 con 0x7f09c40b17d0 2024-03-29T10:50:31.666 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:31.672+0000 7f09c17fa640 10 monclient(hunting): _negotiate 2024-03-29T10:50:31.666 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:31.672+0000 7f09c17fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:50:31.666 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:31.672+0000 7f09c17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:50:31.666 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:31.672+0000 7f09c17fa640 10 monclient(hunting): my global_id is 4619 2024-03-29T10:50:31.667 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:31.672+0000 7f09c17fa640 1 -- 172.21.15.88:0/450823292 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f09a4004eb0 con 0x7f09c40b17d0 2024-03-29T10:50:31.667 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:31.673+0000 7f09c17fa640 1 -- 172.21.15.88:0/450823292 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 650482528 0 0) 0x7f09c406f080 con 0x7f09c41422b0 2024-03-29T10:50:31.667 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:31.673+0000 7f09c17fa640 10 monclient(hunting): _negotiate 2024-03-29T10:50:31.667 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:31.673+0000 7f09c17fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:50:31.667 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:31.673+0000 7f09c17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:50:31.667 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:31.673+0000 7f09c17fa640 10 monclient(hunting): my global_id is 4584 2024-03-29T10:50:31.667 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:31.673+0000 7f09c17fa640 1 -- 172.21.15.88:0/450823292 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f09c406de90 con 0x7f09c41422b0 2024-03-29T10:50:31.667 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:31.673+0000 7f09c17fa640 1 -- 172.21.15.88:0/450823292 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2207955129 0 0) 0x7f09a4004eb0 con 0x7f09c40b17d0 2024-03-29T10:50:31.667 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:31.673+0000 7f09c17fa640 1 -- 172.21.15.88:0/450823292 --> v1:172.21.15.88:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f09c406f080 con 0x7f09c40b17d0 2024-03-29T10:50:31.667 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:31.673+0000 7f09c17fa640 1 -- 172.21.15.88:0/450823292 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f09bc001fe0 con 0x7f09c40b17d0 2024-03-29T10:50:31.667 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:31.673+0000 7f09c17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:50:31.667 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:31.673+0000 7f09c17fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:50:31.667 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:31.673+0000 7f09c17fa640 10 monclient(hunting): dump: 2024-03-29T10:50:31.667 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:50:31.668 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:50:31.668 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:50:31.668 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:50:31.668 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:50:31.668 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:50:31.668 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:50:31.668 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:50:31.668 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:50:31.668 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:50:31.668 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:31.673+0000 7f09c17fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:50:31.668 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:31.673+0000 7f09c17fa640 1 -- 172.21.15.88:0/450823292 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 694605060 0 0) 0x7f09c406de90 con 0x7f09c41422b0 2024-03-29T10:50:31.668 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:31.673+0000 7f09c17fa640 1 -- 172.21.15.88:0/450823292 --> v1:172.21.15.88:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f09a4004eb0 con 0x7f09c41422b0 2024-03-29T10:50:31.668 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:31.673+0000 7f09c17fa640 1 -- 172.21.15.88:0/450823292 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f09b4001790 con 0x7f09c41422b0 2024-03-29T10:50:31.668 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:31.674+0000 7f09c17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:50:31.669 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:31.674+0000 7f09c17fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:50:31.669 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:31.674+0000 7f09c17fa640 10 monclient(hunting): dump: 2024-03-29T10:50:31.669 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:50:31.669 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:50:31.669 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:50:31.669 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:50:31.669 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:50:31.669 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:50:31.669 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:50:31.669 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:50:31.669 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:50:31.669 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:50:31.669 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:31.674+0000 7f09c17fa640 10 client.?.objecter ms_handle_connect 0x7f09c4142690 2024-03-29T10:50:31.669 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:31.674+0000 7f09c17fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:50:31.669 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:31.674+0000 7f09c17fa640 1 -- 172.21.15.88:0/450823292 <== mon.2 v1:172.21.15.88:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3100382470 0 0) 0x7f09c406f080 con 0x7f09c40b17d0 2024-03-29T10:50:31.670 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:31.674+0000 7f09c17fa640 1 -- 172.21.15.88:0/450823292 >> v1:172.21.15.196:6789/0 conn(0x7f09c4142690 legacy=0x7f09c41e7070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:31.670 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:31.674+0000 7f09c17fa640 1 -- 172.21.15.88:0/450823292 >> v1:172.21.15.88:6789/0 conn(0x7f09c41422b0 legacy=0x7f09c412ecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:31.670 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:31.674+0000 7f09c17fa640 10 monclient: _finish_hunting 0 2024-03-29T10:50:31.670 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:31.674+0000 7f09c17fa640 1 monclient: found mon.c 2024-03-29T10:50:31.670 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:31.674+0000 7f09c17fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:50:31.670 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:31.674+0000 7f09c17fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-03-29T10:50:31.670 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:31.674+0000 7f09c17fa640 1 -- 172.21.15.88:0/450823292 --> v1:172.21.15.88:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09c41512f0 con 0x7f09c40b17d0 2024-03-29T10:50:31.670 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:31.674+0000 7f09c17fa640 10 monclient: _finish_auth 0 2024-03-29T10:50:31.670 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:31.674+0000 7f09c17fa640 10 monclient: _check_auth_tickets 2024-03-29T10:50:31.670 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:31.674+0000 7f09c17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:50:31.670 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:31.674+0000 7f09caf9d640 5 monclient: authenticate success, global_id 4619 2024-03-29T10:50:31.670 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:31.674+0000 7f09caf9d640 10 monclient: _renew_subs 2024-03-29T10:50:31.670 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:31.674+0000 7f09caf9d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-03-29T10:50:31.670 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:31.674+0000 7f09caf9d640 1 -- 172.21.15.88:0/450823292 --> v1:172.21.15.88:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09c4071eb0 con 0x7f09c40b17d0 2024-03-29T10:50:31.670 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:31.674+0000 7f09caf9d640 10 client.4619.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:50:31.671 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:31.674+0000 7f09caf9d640 10 monclient: _renew_subs 2024-03-29T10:50:31.671 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:31.674+0000 7f09caf9d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-03-29T10:50:31.671 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:31.674+0000 7f09caf9d640 1 -- 172.21.15.88:0/450823292 --> v1:172.21.15.88:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f09c414fb70 con 0x7f09c40b17d0 2024-03-29T10:50:31.671 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:31.674+0000 7f09caf9d640 1 librados: init done 2024-03-29T10:50:31.671 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:31.675+0000 7f09c17fa640 1 -- 172.21.15.88:0/450823292 <== mon.2 v1:172.21.15.88:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f09bc004e10 con 0x7f09c40b17d0 2024-03-29T10:50:31.671 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:31.675+0000 7f09c17fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:50:31.671 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:31.675+0000 7f09c17fa640 1 -- 172.21.15.88:0/450823292 <== mon.2 v1:172.21.15.88:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f09bc0051a0 con 0x7f09c40b17d0 2024-03-29T10:50:31.671 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:31.675+0000 7f09c2ffd640 4 set_mon_vals no callback set 2024-03-29T10:50:31.671 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:31.675+0000 7f09c17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:50:31.671 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:31.675+0000 7f09c17fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:50:31.671 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:31.675+0000 7f09c17fa640 10 monclient: dump: 2024-03-29T10:50:31.671 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:50:31.671 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:50:31.672 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:50:31.672 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:50:31.672 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:50:31.672 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:50:31.672 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:50:31.672 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:50:31.672 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:50:31.672 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:50:31.672 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:31.676+0000 7f09caf9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:50:31.672 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:31.676+0000 7f09caf9d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:50:31.672 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:31.676+0000 7f09caf9d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-03-29T10:50:31.672 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:31.676+0000 7f09caf9d640 1 -- 172.21.15.88:0/450823292 --> v1:172.21.15.88:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09a8005080 con 0x7f09c40b17d0 2024-03-29T10:50:31.673 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:31.678+0000 7f09c17fa640 1 -- 172.21.15.88:0/450823292 <== mon.2 v1:172.21.15.88:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4235629220 0 0) 0x7f09bc005300 con 0x7f09c40b17d0 2024-03-29T10:50:31.673 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:31.678+0000 7f09c17fa640 1 -- 172.21.15.88:0/450823292 <== mon.2 v1:172.21.15.88:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4257673582 0 0) 0x7f09bc085030 con 0x7f09c40b17d0 2024-03-29T10:50:31.673 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:31.678+0000 7f09c17fa640 10 client.4619.objecter ms_dispatch 0x7f09c4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:50:31.673 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:31.678+0000 7f09c17fa640 3 client.4619.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:50:31.673 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:31.678+0000 7f09c17fa640 3 client.4619.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:50:31.673 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:31.678+0000 7f09c17fa640 20 client.4619.objecter dump_active .. 0 homeless 2024-03-29T10:50:31.673 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:31.679+0000 7f09c3fff640 10 monclient: get_auth_request con 0x7f09a406ded0 auth_method 0 2024-03-29T10:50:31.673 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:31.679+0000 7f09c17fa640 10 client.4619.objecter ms_handle_connect 0x7f09a406ded0 2024-03-29T10:50:31.674 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:31.687+0000 7f09c17fa640 1 -- 172.21.15.88:0/450823292 <== mon.2 v1:172.21.15.88:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f09bc050910 con 0x7f09c40b17d0 2024-03-29T10:50:31.674 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:31.687+0000 7f09c17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:50:31.674 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:31.687+0000 7f09c17fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:50:32.042 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:32.059+0000 7f09caf9d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:50:32.042 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:32.059+0000 7f09caf9d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:50:32.042 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:32.060+0000 7f09caf9d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-03-29T10:50:32.042 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:32.060+0000 7f09caf9d640 1 -- 172.21.15.88:0/450823292 --> v1:172.21.15.88:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f09a8005330 con 0x7f09c40b17d0 2024-03-29T10:50:32.042 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:32.060+0000 7f09c17fa640 1 -- 172.21.15.88:0/450823292 <== mon.2 v1:172.21.15.88:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 344506374) 0x7f09bc052840 con 0x7f09c40b17d0 2024-03-29T10:50:32.043 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:32.060+0000 7f09c17fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:50:32.043 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:32.060+0000 7f09c17fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:50:32.043 INFO:teuthology.misc.health.smithi088.stdout: 2024-03-29T10:50:32.043 INFO:teuthology.misc.health.smithi088.stdout:{"epoch":10,"fsid":"869f5188-f92b-42dd-a46d-8c646e0388a9","created":"2024-03-29T10:40:40.285087+0000","modified":"2024-03-29T10:40:42.536230+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:40:42.536230+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a26c0c1c-8de0-438c-a3f6-3df10ea61574","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5de24915-8dec-49b6-8b16-f3170197c591","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ac054af3-d589-4565-9644-e6af75975c53","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3ec4f045-c714-4a22-bd1a-e07330b45ab3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"cddb9867-8e30-4951-9b13-3d1e78b9c45a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dc6d9911-785d-432f-8b53-56a1ccab302f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"77c88437-b587-496e-8b4d-1824ea8b5305","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ce963d96-5a8e-44c3-99de-c0dbebe234ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:50:32.045 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:32.064+0000 7f09caf9d640 10 librados: watch_flush enter 2024-03-29T10:50:32.046 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:32.064+0000 7f09caf9d640 10 librados: watch_flush exit 2024-03-29T10:50:32.046 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:32.064+0000 7f09caf9d640 20 client.4619.objecter shutdown clearing up homeless session... 2024-03-29T10:50:32.046 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:32.064+0000 7f09caf9d640 10 client.4619.objecter successfully canceled tick 2024-03-29T10:50:32.046 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:32.064+0000 7f09caf9d640 1 -- 172.21.15.88:0/450823292 >> v1:172.21.15.88:6816/1752861349 conn(0x7f09a406ded0 legacy=0x7f09a4070390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:32.046 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:32.064+0000 7f09caf9d640 10 monclient: shutdown 2024-03-29T10:50:32.046 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:32.064+0000 7f09caf9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:50:32.046 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:32.064+0000 7f09caf9d640 1 -- 172.21.15.88:0/450823292 >> v1:172.21.15.88:6790/0 conn(0x7f09c40b17d0 legacy=0x7f09c414f280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:32.047 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:32.064+0000 7f09caf9d640 1 -- 172.21.15.88:0/450823292 shutdown_connections 2024-03-29T10:50:32.047 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:32.064+0000 7f09caf9d640 1 -- 172.21.15.88:0/450823292 >> 172.21.15.88:0/450823292 conn(0x7f09c4066a00 msgr2=0x7f09c413f290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:50:32.047 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:32.065+0000 7f09caf9d640 1 -- 172.21.15.88:0/450823292 shutdown_connections 2024-03-29T10:50:32.047 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:32.065+0000 7f09caf9d640 1 -- 172.21.15.88:0/450823292 wait complete. 2024-03-29T10:50:32.047 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:32.065+0000 7f09caf9d640 1 librados: shutdown 2024-03-29T10:50:32.062 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:50:38.869 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:50:39.018 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:39.036+0000 7f2318b8f640 10 monclient: get_monmap_and_config 2024-03-29T10:50:39.018 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:39.036+0000 7f2318b8f640 10 monclient: build_initial_monmap 2024-03-29T10:50:39.019 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:39.036+0000 7f2318b8f640 10 monclient: monmap: 2024-03-29T10:50:39.019 INFO:teuthology.misc.health.smithi088.stderr:epoch 0 2024-03-29T10:50:39.019 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:50:39.019 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:50:39.037873+0000 2024-03-29T10:50:39.019 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:50:39.037873+0000 2024-03-29T10:50:39.019 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 0 (unknown) 2024-03-29T10:50:39.019 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:50:39.019 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-03-29T10:50:39.019 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-29T10:50:39.019 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-b 2024-03-29T10:50:39.019 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:50:39.020 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:39.037+0000 7f2318b8f640 1 Processor -- start 2024-03-29T10:50:39.020 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:39.037+0000 7f2318b8f640 1 -- start start 2024-03-29T10:50:39.020 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:39.037+0000 7f2318b8f640 10 monclient: init 2024-03-29T10:50:39.020 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:39.038+0000 7f2318b8f640 10 monclient: _reopen_session rank -1 2024-03-29T10:50:39.020 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:39.038+0000 7f2318b8f640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:50:39.021 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:39.038+0000 7f2318b8f640 10 monclient(hunting): picked mon.noname-c con 0x7f23140b0770 addr v1:172.21.15.196:6789/0 2024-03-29T10:50:39.021 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:39.038+0000 7f2318b8f640 10 monclient(hunting): picked mon.noname-b con 0x7f231414a700 addr v1:172.21.15.88:6790/0 2024-03-29T10:50:39.021 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:39.038+0000 7f2318b8f640 10 monclient(hunting): picked mon.noname-a con 0x7f23140ac5a0 addr v1:172.21.15.88:6789/0 2024-03-29T10:50:39.021 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:39.038+0000 7f2318b8f640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23140aba70 con 0x7f23140ac5a0 2024-03-29T10:50:39.021 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:39.038+0000 7f2318b8f640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23140aecb0 con 0x7f23140b0770 2024-03-29T10:50:39.021 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:39.038+0000 7f2318b8f640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23140af150 con 0x7f231414a700 2024-03-29T10:50:39.021 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:39.038+0000 7f2318b8f640 10 monclient(hunting): _renew_subs 2024-03-29T10:50:39.021 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:39.038+0000 7f2318b8f640 10 monclient(hunting): authenticate will time out at 1361.274813s 2024-03-29T10:50:39.026 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:39.039+0000 7f2311d9b640 1 --1- >> v1:172.21.15.88:6790/0 conn(0x7f231414a700 0x7f23140abe90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6790/0 says I am v1:172.21.15.88:58026/0 (socket says 172.21.15.88:58026) 2024-03-29T10:50:39.026 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:39.039+0000 7f2311d9b640 1 -- 172.21.15.88:0/1500020942 learned_addr learned my addr 172.21.15.88:0/1500020942 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-29T10:50:39.027 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:39.040+0000 7f231159a640 1 -- 172.21.15.88:0/1500020942 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1688412421 0 0) 0x7f23140af150 con 0x7f231414a700 2024-03-29T10:50:39.027 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:39.040+0000 7f231159a640 10 monclient(hunting): _negotiate 2024-03-29T10:50:39.027 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:39.040+0000 7f231159a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:50:39.027 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:39.040+0000 7f231159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:50:39.027 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:39.040+0000 7f231159a640 10 monclient(hunting): my global_id is 4622 2024-03-29T10:50:39.027 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:39.040+0000 7f231159a640 1 -- 172.21.15.88:0/1500020942 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f22fc003520 con 0x7f231414a700 2024-03-29T10:50:39.027 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:39.042+0000 7f231159a640 1 -- 172.21.15.88:0/1500020942 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1220404773 0 0) 0x7f22fc003520 con 0x7f231414a700 2024-03-29T10:50:39.027 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:39.042+0000 7f231159a640 1 -- 172.21.15.88:0/1500020942 >> v1:172.21.15.196:6789/0 conn(0x7f23140b0770 legacy=0x7f2314149ff0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-29T10:50:39.027 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:39.042+0000 7f231159a640 1 -- 172.21.15.88:0/1500020942 >> v1:172.21.15.88:6789/0 conn(0x7f23140ac5a0 legacy=0x7f2314151f00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:50:39.027 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:39.042+0000 7f231159a640 10 monclient: _finish_hunting 0 2024-03-29T10:50:39.027 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:39.042+0000 7f231159a640 1 monclient: found mon.noname-b 2024-03-29T10:50:39.027 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:39.042+0000 7f231159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:50:39.027 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:39.042+0000 7f231159a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.88:6790/0 2024-03-29T10:50:39.027 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:39.042+0000 7f231159a640 1 -- 172.21.15.88:0/1500020942 --> v1:172.21.15.88:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23140ac9d0 con 0x7f231414a700 2024-03-29T10:50:39.028 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:39.042+0000 7f231159a640 10 monclient: _finish_auth 0 2024-03-29T10:50:39.028 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:39.042+0000 7f231159a640 10 monclient: _check_auth_tickets 2024-03-29T10:50:39.028 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:39.042+0000 7f231159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:50:39.028 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:39.042+0000 7f2318b8f640 5 monclient: authenticate success, global_id 4622 2024-03-29T10:50:39.028 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:39.042+0000 7f2318b8f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:50:39.028 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:39.042+0000 7f231159a640 1 -- 172.21.15.88:0/1500020942 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f2308001690 con 0x7f231414a700 2024-03-29T10:50:39.028 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:39.042+0000 7f231159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:50:39.028 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:39.042+0000 7f231159a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:50:39.028 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:39.042+0000 7f231159a640 10 monclient: dump: 2024-03-29T10:50:39.028 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:50:39.028 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:50:39.028 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:50:39.028 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:50:39.028 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:50:39.029 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:50:39.029 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:50:39.029 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:50:39.029 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:50:39.029 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:50:39.029 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:39.042+0000 7f2318b8f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:50:39.029 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:39.042+0000 7f231159a640 1 -- 172.21.15.88:0/1500020942 <== mon.2 v1:172.21.15.88:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2308001860 con 0x7f231414a700 2024-03-29T10:50:39.029 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:39.042+0000 7f231159a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:50:39.029 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:39.042+0000 7f2318b8f640 10 monclient: get_monmap_and_config success 2024-03-29T10:50:39.029 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:39.042+0000 7f2318b8f640 4 set_mon_vals no callback set 2024-03-29T10:50:39.029 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:39.042+0000 7f231159a640 1 -- 172.21.15.88:0/1500020942 <== mon.2 v1:172.21.15.88:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f2308001cb0 con 0x7f231414a700 2024-03-29T10:50:39.029 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:39.042+0000 7f231159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:50:39.029 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:39.042+0000 7f231159a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:50:39.029 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:39.042+0000 7f231159a640 10 monclient: dump: 2024-03-29T10:50:39.029 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:50:39.030 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:50:39.030 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:50:39.030 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:50:39.030 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:50:39.030 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:50:39.030 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:50:39.030 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:50:39.030 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:50:39.030 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:50:39.030 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:39.043+0000 7f2318b8f640 10 monclient: shutdown 2024-03-29T10:50:39.030 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:39.043+0000 7f2318b8f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:50:39.030 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:39.043+0000 7f2318b8f640 1 -- 172.21.15.88:0/1500020942 >> v1:172.21.15.88:6790/0 conn(0x7f231414a700 legacy=0x7f23140abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:39.030 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:39.044+0000 7f2318b8f640 1 -- 172.21.15.88:0/1500020942 shutdown_connections 2024-03-29T10:50:39.031 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:39.044+0000 7f2318b8f640 1 -- 172.21.15.88:0/1500020942 >> 172.21.15.88:0/1500020942 conn(0x7f231406b660 msgr2=0x7f231406ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:50:39.031 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:39.044+0000 7f2318b8f640 1 -- 172.21.15.88:0/1500020942 shutdown_connections 2024-03-29T10:50:39.031 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:39.045+0000 7f2318b8f640 1 -- 172.21.15.88:0/1500020942 wait complete. 2024-03-29T10:50:39.034 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:39.046+0000 7f2318b8f640 10 monclient: build_initial_monmap 2024-03-29T10:50:39.034 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:39.046+0000 7f2318b8f640 10 monclient: monmap: 2024-03-29T10:50:39.034 INFO:teuthology.misc.health.smithi088.stderr:epoch 0 2024-03-29T10:50:39.034 INFO:teuthology.misc.health.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:50:39.034 INFO:teuthology.misc.health.smithi088.stderr:last_changed 0.000000 2024-03-29T10:50:39.034 INFO:teuthology.misc.health.smithi088.stderr:created 0.000000 2024-03-29T10:50:39.034 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 0 (unknown) 2024-03-29T10:50:39.034 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:50:39.034 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-03-29T10:50:39.034 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-29T10:50:39.034 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-c 2024-03-29T10:50:39.034 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:50:39.035 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:39.046+0000 7f2318b8f640 1 librados: starting msgr at 2024-03-29T10:50:39.035 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:39.046+0000 7f2318b8f640 1 librados: starting objecter 2024-03-29T10:50:39.035 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:39.046+0000 7f2318b8f640 1 Processor -- start 2024-03-29T10:50:39.035 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:39.046+0000 7f2318b8f640 1 -- start start 2024-03-29T10:50:39.035 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:39.046+0000 7f2318b8f640 1 librados: setting wanted keys 2024-03-29T10:50:39.035 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:39.046+0000 7f2318b8f640 1 librados: calling monclient init 2024-03-29T10:50:39.035 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:39.046+0000 7f2318b8f640 10 monclient: init 2024-03-29T10:50:39.035 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:39.048+0000 7f2318b8f640 10 monclient: _reopen_session rank -1 2024-03-29T10:50:39.035 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:39.048+0000 7f2318b8f640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:50:39.035 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:39.048+0000 7f2318b8f640 10 monclient(hunting): picked mon.noname-c con 0x7f23140ac5a0 addr v1:172.21.15.88:6790/0 2024-03-29T10:50:39.035 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:39.048+0000 7f231259c640 1 --1- >> v1:172.21.15.88:6790/0 conn(0x7f23140ac5a0 0x7f23141285a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6790/0 says I am v1:172.21.15.88:33268/0 (socket says 172.21.15.88:33268) 2024-03-29T10:50:39.035 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:39.048+0000 7f231259c640 1 -- 172.21.15.88:0/4010530308 learned_addr learned my addr 172.21.15.88:0/4010530308 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-29T10:50:39.035 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:39.048+0000 7f2318b8f640 10 monclient(hunting): picked mon.noname-b con 0x7f23140b0770 addr v1:172.21.15.196:6789/0 2024-03-29T10:50:39.035 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:39.048+0000 7f2318b8f640 10 monclient(hunting): picked mon.noname-a con 0x7f231414a700 addr v1:172.21.15.88:6789/0 2024-03-29T10:50:39.035 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:39.048+0000 7f2318b8f640 1 -- 172.21.15.88:0/4010530308 --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2314128110 con 0x7f231414a700 2024-03-29T10:50:39.036 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:39.048+0000 7f2318b8f640 1 -- 172.21.15.88:0/4010530308 --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f231412a400 con 0x7f23140b0770 2024-03-29T10:50:39.036 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:39.048+0000 7f2318b8f640 1 -- 172.21.15.88:0/4010530308 --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2314127da0 con 0x7f23140ac5a0 2024-03-29T10:50:39.036 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:39.049+0000 7f2318b8f640 10 monclient(hunting): _renew_subs 2024-03-29T10:50:39.036 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:39.049+0000 7f2318b8f640 10 monclient(hunting): authenticate will time out at 1361.285023s 2024-03-29T10:50:39.036 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:39.049+0000 7f2302ffd640 10 client.?.objecter ms_handle_connect 0x7f23140ac5a0 2024-03-29T10:50:39.036 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:39.049+0000 7f2302ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:50:39.036 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:39.051+0000 7f2302ffd640 1 -- 172.21.15.88:0/4010530308 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3470213949 0 0) 0x7f2314127da0 con 0x7f23140ac5a0 2024-03-29T10:50:39.036 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:39.051+0000 7f2302ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:50:39.036 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:39.051+0000 7f2302ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:50:39.036 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:39.051+0000 7f2302ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:50:39.036 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:39.051+0000 7f2302ffd640 10 monclient(hunting): my global_id is 4625 2024-03-29T10:50:39.036 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:39.051+0000 7f2302ffd640 1 -- 172.21.15.88:0/4010530308 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f22f4004eb0 con 0x7f23140ac5a0 2024-03-29T10:50:39.036 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:39.052+0000 7f2302ffd640 10 client.?.objecter ms_handle_connect 0x7f23140b0770 2024-03-29T10:50:39.036 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:39.052+0000 7f2302ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:50:39.036 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:39.052+0000 7f2302ffd640 1 -- 172.21.15.88:0/4010530308 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1801488951 0 0) 0x7f231412a400 con 0x7f23140b0770 2024-03-29T10:50:39.037 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:39.052+0000 7f2302ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:50:39.037 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:39.052+0000 7f2302ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:50:39.037 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:39.052+0000 7f2302ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:50:39.037 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:39.052+0000 7f2302ffd640 10 monclient(hunting): my global_id is 4567 2024-03-29T10:50:39.037 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:39.052+0000 7f2302ffd640 1 -- 172.21.15.88:0/4010530308 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2314127da0 con 0x7f23140b0770 2024-03-29T10:50:39.040 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:39.053+0000 7f2302ffd640 1 -- 172.21.15.88:0/4010530308 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2172363069 0 0) 0x7f22f4004eb0 con 0x7f23140ac5a0 2024-03-29T10:50:39.040 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:39.053+0000 7f2302ffd640 1 -- 172.21.15.88:0/4010530308 --> v1:172.21.15.88:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f231412a400 con 0x7f23140ac5a0 2024-03-29T10:50:39.040 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:39.053+0000 7f2302ffd640 1 -- 172.21.15.88:0/4010530308 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f2304001870 con 0x7f23140ac5a0 2024-03-29T10:50:39.040 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:39.053+0000 7f2302ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:50:39.040 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:39.054+0000 7f2302ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:50:39.040 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:39.054+0000 7f2302ffd640 10 monclient(hunting): dump: 2024-03-29T10:50:39.040 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:50:39.040 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:50:39.040 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:50:39.040 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:50:39.040 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:50:39.040 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:50:39.041 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:50:39.041 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:50:39.041 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:50:39.041 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:50:39.041 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:39.054+0000 7f2302ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:50:39.041 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:39.054+0000 7f2302ffd640 1 -- 172.21.15.88:0/4010530308 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 540479501 0 0) 0x7f2314127da0 con 0x7f23140b0770 2024-03-29T10:50:39.041 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:39.054+0000 7f2302ffd640 1 -- 172.21.15.88:0/4010530308 --> v1:172.21.15.196:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f22f4004eb0 con 0x7f23140b0770 2024-03-29T10:50:39.041 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:39.054+0000 7f2302ffd640 1 -- 172.21.15.88:0/4010530308 <== mon.2 v1:172.21.15.88:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1446192378 0 0) 0x7f231412a400 con 0x7f23140ac5a0 2024-03-29T10:50:39.041 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:39.054+0000 7f2302ffd640 1 -- 172.21.15.88:0/4010530308 >> v1:172.21.15.196:6789/0 conn(0x7f23140b0770 legacy=0x7f2314128cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:39.041 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:39.054+0000 7f2302ffd640 1 -- 172.21.15.88:0/4010530308 >> v1:172.21.15.88:6789/0 conn(0x7f231414a700 legacy=0x7f23141e5ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:39.041 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:39.054+0000 7f2302ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:50:39.041 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:39.054+0000 7f2302ffd640 1 monclient: found mon.c 2024-03-29T10:50:39.041 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:39.054+0000 7f2302ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:50:39.042 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:39.054+0000 7f2302ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-03-29T10:50:39.042 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:39.054+0000 7f2302ffd640 1 -- 172.21.15.88:0/4010530308 --> v1:172.21.15.88:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2314147bb0 con 0x7f23140ac5a0 2024-03-29T10:50:39.042 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:39.054+0000 7f2302ffd640 10 monclient: _finish_auth 0 2024-03-29T10:50:39.042 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:39.055+0000 7f2302ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:50:39.042 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:39.055+0000 7f2302ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:50:39.042 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:39.055+0000 7f2302ffd640 1 -- 172.21.15.88:0/4010530308 <== mon.2 v1:172.21.15.88:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f23040019d0 con 0x7f23140ac5a0 2024-03-29T10:50:39.042 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:39.055+0000 7f2302ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:50:39.042 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:39.055+0000 7f2302ffd640 1 -- 172.21.15.88:0/4010530308 <== mon.2 v1:172.21.15.88:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f2304004aa0 con 0x7f23140ac5a0 2024-03-29T10:50:39.042 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:39.055+0000 7f2302ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:50:39.042 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:39.055+0000 7f2302ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:50:39.042 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:39.055+0000 7f2303fff640 4 set_mon_vals no callback set 2024-03-29T10:50:39.042 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:39.055+0000 7f2302ffd640 10 monclient: dump: 2024-03-29T10:50:39.042 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:50:39.042 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:50:39.043 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:50:39.043 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:50:39.043 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:50:39.043 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:50:39.043 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:50:39.043 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:50:39.043 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:50:39.043 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:50:39.043 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:39.055+0000 7f2318b8f640 5 monclient: authenticate success, global_id 4625 2024-03-29T10:50:39.043 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:39.055+0000 7f2318b8f640 10 monclient: _renew_subs 2024-03-29T10:50:39.043 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:39.055+0000 7f2318b8f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-03-29T10:50:39.043 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:39.055+0000 7f2318b8f640 1 -- 172.21.15.88:0/4010530308 --> v1:172.21.15.88:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23141df630 con 0x7f23140ac5a0 2024-03-29T10:50:39.043 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:39.055+0000 7f2318b8f640 10 client.4625.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:50:39.043 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:39.055+0000 7f2318b8f640 10 monclient: _renew_subs 2024-03-29T10:50:39.043 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:39.055+0000 7f2318b8f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-03-29T10:50:39.044 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:39.055+0000 7f2318b8f640 1 -- 172.21.15.88:0/4010530308 --> v1:172.21.15.88:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f23141deda0 con 0x7f23140ac5a0 2024-03-29T10:50:39.044 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:39.055+0000 7f2318b8f640 1 librados: init done 2024-03-29T10:50:39.044 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:39.057+0000 7f2318b8f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:50:39.044 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:39.060+0000 7f2318b8f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:50:39.044 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:39.060+0000 7f2318b8f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-03-29T10:50:39.045 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:39.060+0000 7f2318b8f640 1 -- 172.21.15.88:0/4010530308 --> v1:172.21.15.88:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2314075f10 con 0x7f23140ac5a0 2024-03-29T10:50:39.045 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:39.060+0000 7f2302ffd640 1 -- 172.21.15.88:0/4010530308 <== mon.2 v1:172.21.15.88:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4235629220 0 0) 0x7f230401aa70 con 0x7f23140ac5a0 2024-03-29T10:50:39.045 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:39.061+0000 7f2302ffd640 1 -- 172.21.15.88:0/4010530308 <== mon.2 v1:172.21.15.88:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4257673582 0 0) 0x7f2304084a00 con 0x7f23140ac5a0 2024-03-29T10:50:39.045 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:39.061+0000 7f2302ffd640 10 client.4625.objecter ms_dispatch 0x7f2314000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:50:39.045 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:39.061+0000 7f2302ffd640 3 client.4625.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:50:39.045 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:39.061+0000 7f2302ffd640 3 client.4625.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:50:39.045 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:39.061+0000 7f2311d9b640 10 monclient: get_auth_request con 0x7f22f406dea0 auth_method 0 2024-03-29T10:50:39.045 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:39.061+0000 7f2302ffd640 20 client.4625.objecter dump_active .. 0 homeless 2024-03-29T10:50:39.045 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:39.062+0000 7f2302ffd640 10 client.4625.objecter ms_handle_connect 0x7f22f406dea0 2024-03-29T10:50:39.051 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:39.069+0000 7f2302ffd640 1 -- 172.21.15.88:0/4010530308 <== mon.2 v1:172.21.15.88:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f23040502e0 con 0x7f23140ac5a0 2024-03-29T10:50:39.051 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:39.070+0000 7f2302ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:50:39.051 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:39.070+0000 7f2302ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:50:39.373 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:39.391+0000 7f2318b8f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:50:39.373 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:39.391+0000 7f2318b8f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:50:39.373 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:39.391+0000 7f2318b8f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-03-29T10:50:39.373 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:39.391+0000 7f2318b8f640 1 -- 172.21.15.88:0/4010530308 --> v1:172.21.15.88:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2314147d40 con 0x7f23140ac5a0 2024-03-29T10:50:39.374 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:39.392+0000 7f2302ffd640 1 -- 172.21.15.88:0/4010530308 <== mon.2 v1:172.21.15.88:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 344506374) 0x7f2304052210 con 0x7f23140ac5a0 2024-03-29T10:50:39.374 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:39.392+0000 7f2302ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:50:39.374 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:39.392+0000 7f2302ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:50:39.376 INFO:teuthology.misc.health.smithi088.stdout: 2024-03-29T10:50:39.376 INFO:teuthology.misc.health.smithi088.stdout:{"epoch":10,"fsid":"869f5188-f92b-42dd-a46d-8c646e0388a9","created":"2024-03-29T10:40:40.285087+0000","modified":"2024-03-29T10:40:42.536230+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:40:42.536230+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a26c0c1c-8de0-438c-a3f6-3df10ea61574","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5de24915-8dec-49b6-8b16-f3170197c591","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ac054af3-d589-4565-9644-e6af75975c53","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3ec4f045-c714-4a22-bd1a-e07330b45ab3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"cddb9867-8e30-4951-9b13-3d1e78b9c45a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dc6d9911-785d-432f-8b53-56a1ccab302f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"77c88437-b587-496e-8b4d-1824ea8b5305","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ce963d96-5a8e-44c3-99de-c0dbebe234ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:50:39.380 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:39.397+0000 7f2318b8f640 10 librados: watch_flush enter 2024-03-29T10:50:39.380 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:39.397+0000 7f2318b8f640 10 librados: watch_flush exit 2024-03-29T10:50:39.380 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:39.397+0000 7f2318b8f640 20 client.4625.objecter shutdown clearing up homeless session... 2024-03-29T10:50:39.380 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:39.397+0000 7f2318b8f640 10 client.4625.objecter successfully canceled tick 2024-03-29T10:50:39.380 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:39.398+0000 7f2318b8f640 1 -- 172.21.15.88:0/4010530308 >> v1:172.21.15.88:6816/1752861349 conn(0x7f22f406dea0 legacy=0x7f22f4070360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:39.380 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:39.398+0000 7f2318b8f640 10 monclient: shutdown 2024-03-29T10:50:39.380 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:39.398+0000 7f2318b8f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:50:39.380 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:39.398+0000 7f2318b8f640 1 -- 172.21.15.88:0/4010530308 >> v1:172.21.15.88:6790/0 conn(0x7f23140ac5a0 legacy=0x7f23141285a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:39.381 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:39.398+0000 7f2318b8f640 1 -- 172.21.15.88:0/4010530308 shutdown_connections 2024-03-29T10:50:39.381 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:39.398+0000 7f2318b8f640 1 -- 172.21.15.88:0/4010530308 >> 172.21.15.88:0/4010530308 conn(0x7f231406b660 msgr2=0x7f231406ba40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:50:39.381 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:39.398+0000 7f2318b8f640 1 -- 172.21.15.88:0/4010530308 shutdown_connections 2024-03-29T10:50:39.381 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:39.398+0000 7f2318b8f640 1 -- 172.21.15.88:0/4010530308 wait complete. 2024-03-29T10:50:39.381 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:39.398+0000 7f2318b8f640 1 librados: shutdown 2024-03-29T10:50:39.399 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:50:46.206 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:50:46.351 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:46.369+0000 7f9cede25640 10 monclient: get_monmap_and_config 2024-03-29T10:50:46.351 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:46.369+0000 7f9cede25640 10 monclient: build_initial_monmap 2024-03-29T10:50:46.351 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:46.369+0000 7f9cede25640 10 monclient: monmap: 2024-03-29T10:50:46.351 INFO:teuthology.misc.health.smithi088.stderr:epoch 0 2024-03-29T10:50:46.351 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:50:46.351 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:50:46.370431+0000 2024-03-29T10:50:46.351 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:50:46.370431+0000 2024-03-29T10:50:46.351 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 0 (unknown) 2024-03-29T10:50:46.352 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:50:46.352 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-03-29T10:50:46.352 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-29T10:50:46.352 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-b 2024-03-29T10:50:46.352 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:50:46.354 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:46.373+0000 7f9cede25640 1 Processor -- start 2024-03-29T10:50:46.354 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:46.373+0000 7f9cede25640 1 -- start start 2024-03-29T10:50:46.355 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:46.373+0000 7f9cede25640 10 monclient: init 2024-03-29T10:50:46.356 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:46.374+0000 7f9cede25640 10 monclient: _reopen_session rank -1 2024-03-29T10:50:46.356 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:46.374+0000 7f9cede25640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:50:46.356 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:46.374+0000 7f9cede25640 10 monclient(hunting): picked mon.noname-b con 0x7f9ce81422b0 addr v1:172.21.15.88:6790/0 2024-03-29T10:50:46.356 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:46.374+0000 7f9cede25640 10 monclient(hunting): picked mon.noname-a con 0x7f9ce8142690 addr v1:172.21.15.88:6789/0 2024-03-29T10:50:46.356 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:46.374+0000 7f9cede25640 10 monclient(hunting): picked mon.noname-c con 0x7f9ce80b17d0 addr v1:172.21.15.196:6789/0 2024-03-29T10:50:46.356 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:46.374+0000 7f9cede25640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ce8142c80 con 0x7f9ce8142690 2024-03-29T10:50:46.356 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:46.374+0000 7f9cede25640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ce80aae40 con 0x7f9ce80b17d0 2024-03-29T10:50:46.356 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:46.374+0000 7f9cede25640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ce813fbd0 con 0x7f9ce81422b0 2024-03-29T10:50:46.356 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:46.374+0000 7f9cede25640 10 monclient(hunting): _renew_subs 2024-03-29T10:50:46.356 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:46.374+0000 7f9cede25640 10 monclient(hunting): authenticate will time out at 1368.610180s 2024-03-29T10:50:46.356 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:46.374+0000 7f9ce77fe640 1 --1- >> v1:172.21.15.88:6790/0 conn(0x7f9ce81422b0 0x7f9ce813f4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6790/0 says I am v1:172.21.15.88:35030/0 (socket says 172.21.15.88:35030) 2024-03-29T10:50:46.356 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:46.374+0000 7f9ce77fe640 1 -- 172.21.15.88:0/2013502304 learned_addr learned my addr 172.21.15.88:0/2013502304 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-29T10:50:46.357 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:46.374+0000 7f9ce67fc640 1 -- 172.21.15.88:0/2013502304 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 842804527 0 0) 0x7f9ce813fbd0 con 0x7f9ce81422b0 2024-03-29T10:50:46.357 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:46.374+0000 7f9ce67fc640 10 monclient(hunting): _negotiate 2024-03-29T10:50:46.357 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:46.374+0000 7f9ce67fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:50:46.357 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:46.374+0000 7f9ce67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:50:46.357 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:46.374+0000 7f9ce67fc640 10 monclient(hunting): my global_id is 4628 2024-03-29T10:50:46.357 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:46.374+0000 7f9ce67fc640 1 -- 172.21.15.88:0/2013502304 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9cd0003520 con 0x7f9ce81422b0 2024-03-29T10:50:46.357 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:46.375+0000 7f9ce67fc640 1 -- 172.21.15.88:0/2013502304 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 527636580 0 0) 0x7f9cd0003520 con 0x7f9ce81422b0 2024-03-29T10:50:46.358 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:46.375+0000 7f9ce67fc640 1 -- 172.21.15.88:0/2013502304 >> v1:172.21.15.196:6789/0 conn(0x7f9ce80b17d0 legacy=0x7f9ce80b2ca0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:50:46.360 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:46.377+0000 7f9ce67fc640 1 -- 172.21.15.88:0/2013502304 >> v1:172.21.15.88:6789/0 conn(0x7f9ce8142690 legacy=0x7f9ce80b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:50:46.360 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:46.378+0000 7f9ce67fc640 10 monclient: _finish_hunting 0 2024-03-29T10:50:46.361 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:46.378+0000 7f9ce67fc640 1 monclient: found mon.noname-b 2024-03-29T10:50:46.361 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:46.378+0000 7f9ce67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:50:46.361 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:46.378+0000 7f9ce67fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.88:6790/0 2024-03-29T10:50:46.361 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:46.378+0000 7f9ce67fc640 1 -- 172.21.15.88:0/2013502304 --> v1:172.21.15.88:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ce8151940 con 0x7f9ce81422b0 2024-03-29T10:50:46.361 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:46.378+0000 7f9ce67fc640 10 monclient: _finish_auth 0 2024-03-29T10:50:46.361 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:46.378+0000 7f9ce67fc640 10 monclient: _check_auth_tickets 2024-03-29T10:50:46.361 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:46.378+0000 7f9ce67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:50:46.361 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:46.378+0000 7f9ce67fc640 1 -- 172.21.15.88:0/2013502304 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f9cdc001690 con 0x7f9ce81422b0 2024-03-29T10:50:46.361 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:46.378+0000 7f9ce67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:50:46.361 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:46.378+0000 7f9ce67fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:50:46.361 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:46.378+0000 7f9ce67fc640 10 monclient: dump: 2024-03-29T10:50:46.361 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:50:46.361 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:50:46.362 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:50:46.362 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:50:46.362 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:50:46.362 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:50:46.362 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:50:46.362 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:50:46.362 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:50:46.362 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:50:46.362 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:46.378+0000 7f9cede25640 5 monclient: authenticate success, global_id 4628 2024-03-29T10:50:46.362 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:46.378+0000 7f9cede25640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:50:46.363 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:46.379+0000 7f9ce67fc640 1 -- 172.21.15.88:0/2013502304 <== mon.2 v1:172.21.15.88:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9cdc0017f0 con 0x7f9ce81422b0 2024-03-29T10:50:46.364 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:46.379+0000 7f9ce67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:50:46.364 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:46.379+0000 7f9ce67fc640 1 -- 172.21.15.88:0/2013502304 <== mon.2 v1:172.21.15.88:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f9cdc001c40 con 0x7f9ce81422b0 2024-03-29T10:50:46.364 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:46.379+0000 7f9ce67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:50:46.364 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:46.380+0000 7f9ce67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:50:46.364 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:46.380+0000 7f9ce67fc640 10 monclient: dump: 2024-03-29T10:50:46.364 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:50:46.364 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:50:46.364 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:50:46.364 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:50:46.364 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:50:46.364 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:50:46.364 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:50:46.364 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:50:46.364 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:50:46.365 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:50:46.365 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:46.380+0000 7f9cede25640 10 monclient: get_monmap_and_config success 2024-03-29T10:50:46.365 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:46.380+0000 7f9cede25640 4 set_mon_vals no callback set 2024-03-29T10:50:46.365 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:46.381+0000 7f9cede25640 10 monclient: shutdown 2024-03-29T10:50:46.365 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:46.381+0000 7f9cede25640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:50:46.365 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:46.381+0000 7f9cede25640 1 -- 172.21.15.88:0/2013502304 >> v1:172.21.15.88:6790/0 conn(0x7f9ce81422b0 legacy=0x7f9ce813f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:46.365 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:46.381+0000 7f9cede25640 1 -- 172.21.15.88:0/2013502304 shutdown_connections 2024-03-29T10:50:46.365 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:46.381+0000 7f9cede25640 1 -- 172.21.15.88:0/2013502304 >> 172.21.15.88:0/2013502304 conn(0x7f9ce8066a00 msgr2=0x7f9ce8066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:50:46.365 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:46.384+0000 7f9cede25640 1 -- 172.21.15.88:0/2013502304 shutdown_connections 2024-03-29T10:50:46.365 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:46.384+0000 7f9cede25640 1 -- 172.21.15.88:0/2013502304 wait complete. 2024-03-29T10:50:46.367 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:46.385+0000 7f9cede25640 10 monclient: build_initial_monmap 2024-03-29T10:50:46.368 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:46.385+0000 7f9cede25640 10 monclient: monmap: 2024-03-29T10:50:46.368 INFO:teuthology.misc.health.smithi088.stderr:epoch 0 2024-03-29T10:50:46.368 INFO:teuthology.misc.health.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:50:46.369 INFO:teuthology.misc.health.smithi088.stderr:last_changed 0.000000 2024-03-29T10:50:46.369 INFO:teuthology.misc.health.smithi088.stderr:created 0.000000 2024-03-29T10:50:46.369 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 0 (unknown) 2024-03-29T10:50:46.369 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:50:46.369 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-03-29T10:50:46.369 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-29T10:50:46.369 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-c 2024-03-29T10:50:46.369 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:50:46.369 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:46.385+0000 7f9cede25640 1 librados: starting msgr at 2024-03-29T10:50:46.370 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:46.385+0000 7f9cede25640 1 librados: starting objecter 2024-03-29T10:50:46.374 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:46.385+0000 7f9cede25640 1 Processor -- start 2024-03-29T10:50:46.374 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:46.386+0000 7f9cede25640 1 -- start start 2024-03-29T10:50:46.374 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:46.386+0000 7f9cede25640 1 librados: setting wanted keys 2024-03-29T10:50:46.374 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:46.386+0000 7f9cede25640 1 librados: calling monclient init 2024-03-29T10:50:46.374 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:46.386+0000 7f9cede25640 10 monclient: init 2024-03-29T10:50:46.374 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:46.386+0000 7f9cede25640 10 monclient: _reopen_session rank -1 2024-03-29T10:50:46.374 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:46.386+0000 7f9cede25640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:50:46.375 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:46.386+0000 7f9cede25640 10 monclient(hunting): picked mon.noname-c con 0x7f9ce80b17d0 addr v1:172.21.15.88:6790/0 2024-03-29T10:50:46.375 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:46.386+0000 7f9cede25640 10 monclient(hunting): picked mon.noname-b con 0x7f9ce81422b0 addr v1:172.21.15.196:6789/0 2024-03-29T10:50:46.375 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:46.386+0000 7f9cede25640 10 monclient(hunting): picked mon.noname-a con 0x7f9ce8142690 addr v1:172.21.15.88:6789/0 2024-03-29T10:50:46.375 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:46.386+0000 7f9cede25640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ce806eb50 con 0x7f9ce8142690 2024-03-29T10:50:46.375 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:46.386+0000 7f9cede25640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ce806eda0 con 0x7f9ce81422b0 2024-03-29T10:50:46.375 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:46.386+0000 7f9cede25640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ce806f080 con 0x7f9ce80b17d0 2024-03-29T10:50:46.375 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:46.386+0000 7f9cede25640 10 monclient(hunting): _renew_subs 2024-03-29T10:50:46.375 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:46.386+0000 7f9cede25640 10 monclient(hunting): authenticate will time out at 1368.622730s 2024-03-29T10:50:46.375 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:46.386+0000 7f9ce77fe640 1 --1- >> v1:172.21.15.88:6790/0 conn(0x7f9ce80b17d0 0x7f9ce812ed40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6790/0 says I am v1:172.21.15.88:40086/0 (socket says 172.21.15.88:40086) 2024-03-29T10:50:46.375 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:46.387+0000 7f9ce77fe640 1 -- 172.21.15.88:0/353643557 learned_addr learned my addr 172.21.15.88:0/353643557 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-29T10:50:46.375 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:46.387+0000 7f9ccffff640 10 client.?.objecter ms_handle_connect 0x7f9ce80b17d0 2024-03-29T10:50:46.375 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:46.387+0000 7f9ccffff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:50:46.375 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:46.387+0000 7f9ccffff640 1 -- 172.21.15.88:0/353643557 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3406971816 0 0) 0x7f9ce806f080 con 0x7f9ce80b17d0 2024-03-29T10:50:46.375 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:46.387+0000 7f9ccffff640 10 monclient(hunting): _negotiate 2024-03-29T10:50:46.376 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:46.387+0000 7f9ccffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:50:46.376 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:46.387+0000 7f9ccffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:50:46.376 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:46.387+0000 7f9ccffff640 10 monclient(hunting): my global_id is 4631 2024-03-29T10:50:46.376 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:46.387+0000 7f9ccffff640 1 -- 172.21.15.88:0/353643557 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9cd4004eb0 con 0x7f9ce80b17d0 2024-03-29T10:50:46.376 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:46.389+0000 7f9ccffff640 1 -- 172.21.15.88:0/353643557 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3307648001 0 0) 0x7f9cd4004eb0 con 0x7f9ce80b17d0 2024-03-29T10:50:46.376 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:46.389+0000 7f9ccffff640 1 -- 172.21.15.88:0/353643557 --> v1:172.21.15.88:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9ce806f080 con 0x7f9ce80b17d0 2024-03-29T10:50:46.376 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:46.389+0000 7f9ccffff640 1 -- 172.21.15.88:0/353643557 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f9cdc001bc0 con 0x7f9ce80b17d0 2024-03-29T10:50:46.376 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:46.389+0000 7f9ccffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:50:46.376 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:46.389+0000 7f9ccffff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:50:46.376 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:46.389+0000 7f9ccffff640 10 monclient(hunting): dump: 2024-03-29T10:50:46.376 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:50:46.376 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:50:46.376 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:50:46.376 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:50:46.377 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:50:46.377 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:50:46.377 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:50:46.377 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:50:46.377 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:50:46.377 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:50:46.377 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:46.389+0000 7f9ccffff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:50:46.377 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:46.389+0000 7f9ccffff640 1 -- 172.21.15.88:0/353643557 <== mon.2 v1:172.21.15.88:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 13549246 0 0) 0x7f9ce806f080 con 0x7f9ce80b17d0 2024-03-29T10:50:46.377 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:46.389+0000 7f9ccffff640 1 -- 172.21.15.88:0/353643557 >> v1:172.21.15.196:6789/0 conn(0x7f9ce81422b0 legacy=0x7f9ce81db910 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:50:46.377 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:46.390+0000 7f9ccffff640 1 -- 172.21.15.88:0/353643557 >> v1:172.21.15.88:6789/0 conn(0x7f9ce8142690 legacy=0x7f9ce81e7150 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:50:46.381 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:46.393+0000 7f9ccffff640 10 monclient: _finish_hunting 0 2024-03-29T10:50:46.381 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:46.393+0000 7f9ccffff640 1 monclient: found mon.c 2024-03-29T10:50:46.381 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:46.393+0000 7f9ccffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:50:46.382 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:46.393+0000 7f9ccffff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-03-29T10:50:46.382 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:46.393+0000 7f9ccffff640 1 -- 172.21.15.88:0/353643557 --> v1:172.21.15.88:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ce8131680 con 0x7f9ce80b17d0 2024-03-29T10:50:46.382 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:46.393+0000 7f9ccffff640 10 monclient: _finish_auth 0 2024-03-29T10:50:46.382 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:46.393+0000 7f9ccffff640 10 monclient: _check_auth_tickets 2024-03-29T10:50:46.382 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:46.393+0000 7f9ccffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:50:46.382 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:46.393+0000 7f9cede25640 5 monclient: authenticate success, global_id 4631 2024-03-29T10:50:46.382 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:46.393+0000 7f9cede25640 10 monclient: _renew_subs 2024-03-29T10:50:46.382 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:46.393+0000 7f9cede25640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-03-29T10:50:46.382 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:46.393+0000 7f9cede25640 1 -- 172.21.15.88:0/353643557 --> v1:172.21.15.88:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ce812f450 con 0x7f9ce80b17d0 2024-03-29T10:50:46.382 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:46.393+0000 7f9cede25640 10 client.4631.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:50:46.382 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:46.393+0000 7f9cede25640 10 monclient: _renew_subs 2024-03-29T10:50:46.382 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:46.393+0000 7f9cede25640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-03-29T10:50:46.383 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:46.393+0000 7f9cede25640 1 -- 172.21.15.88:0/353643557 --> v1:172.21.15.88:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ce806e070 con 0x7f9ce80b17d0 2024-03-29T10:50:46.383 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:46.393+0000 7f9cede25640 1 librados: init done 2024-03-29T10:50:46.383 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:46.394+0000 7f9ccffff640 1 -- 172.21.15.88:0/353643557 <== mon.2 v1:172.21.15.88:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9cdc001d20 con 0x7f9ce80b17d0 2024-03-29T10:50:46.383 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:46.394+0000 7f9ccffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:50:46.383 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:46.394+0000 7f9ccffff640 1 -- 172.21.15.88:0/353643557 <== mon.2 v1:172.21.15.88:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f9cdc004d90 con 0x7f9ce80b17d0 2024-03-29T10:50:46.383 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:46.394+0000 7f9ccffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:50:46.383 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:46.394+0000 7f9ccffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:50:46.383 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:46.394+0000 7f9ccffff640 10 monclient: dump: 2024-03-29T10:50:46.383 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:50:46.383 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:50:46.383 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:50:46.383 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:50:46.384 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:50:46.384 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:50:46.384 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:50:46.384 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:50:46.384 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:50:46.384 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:50:46.384 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:46.394+0000 7f9ce5ffb640 4 set_mon_vals no callback set 2024-03-29T10:50:46.384 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:46.396+0000 7f9ccffff640 1 -- 172.21.15.88:0/353643557 <== mon.2 v1:172.21.15.88:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4235629220 0 0) 0x7f9cdc004ef0 con 0x7f9ce80b17d0 2024-03-29T10:50:46.385 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:46.396+0000 7f9cede25640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:50:46.385 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:46.396+0000 7f9cede25640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:50:46.385 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:46.396+0000 7f9cede25640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-03-29T10:50:46.385 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:46.396+0000 7f9cede25640 1 -- 172.21.15.88:0/353643557 --> v1:172.21.15.88:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9cc8005080 con 0x7f9ce80b17d0 2024-03-29T10:50:46.385 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:46.396+0000 7f9ccffff640 1 -- 172.21.15.88:0/353643557 <== mon.2 v1:172.21.15.88:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1407902845 0 0) 0x7f9cdc095d80 con 0x7f9ce80b17d0 2024-03-29T10:50:46.385 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:46.396+0000 7f9ccffff640 10 client.4631.objecter ms_dispatch 0x7f9ce8000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T10:50:46.385 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:46.397+0000 7f9ccffff640 3 client.4631.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T10:50:46.385 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:46.397+0000 7f9ccffff640 3 client.4631.objecter handle_osd_map decoding full epoch 11 2024-03-29T10:50:46.385 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:46.397+0000 7f9ccffff640 20 client.4631.objecter dump_active .. 0 homeless 2024-03-29T10:50:46.385 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:46.397+0000 7f9ce6ffd640 10 monclient: get_auth_request con 0x7f9cd406dcb0 auth_method 0 2024-03-29T10:50:46.386 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:46.397+0000 7f9ccffff640 10 client.4631.objecter ms_handle_connect 0x7f9cd406dcb0 2024-03-29T10:50:46.390 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:46.409+0000 7f9ccffff640 1 -- 172.21.15.88:0/353643557 <== mon.2 v1:172.21.15.88:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9cdc061660 con 0x7f9ce80b17d0 2024-03-29T10:50:46.390 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:46.409+0000 7f9ccffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:50:46.390 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:46.409+0000 7f9ccffff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:50:46.720 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:46.738+0000 7f9cede25640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:50:46.720 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:46.738+0000 7f9cede25640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:50:46.720 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:46.738+0000 7f9cede25640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-03-29T10:50:46.721 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:46.738+0000 7f9cede25640 1 -- 172.21.15.88:0/353643557 --> v1:172.21.15.88:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9cc8005330 con 0x7f9ce80b17d0 2024-03-29T10:50:46.721 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:46.739+0000 7f9ccffff640 1 -- 172.21.15.88:0/353643557 <== mon.2 v1:172.21.15.88:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 1123663608) 0x7f9cdc063590 con 0x7f9ce80b17d0 2024-03-29T10:50:46.722 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:46.739+0000 7f9ccffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:50:46.722 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:46.739+0000 7f9ccffff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:50:46.722 INFO:teuthology.misc.health.smithi088.stdout: 2024-03-29T10:50:46.722 INFO:teuthology.misc.health.smithi088.stdout:{"epoch":11,"fsid":"869f5188-f92b-42dd-a46d-8c646e0388a9","created":"2024-03-29T10:40:40.285087+0000","modified":"2024-03-29T10:50:45.236723+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:50:45.236723+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a26c0c1c-8de0-438c-a3f6-3df10ea61574","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"5de24915-8dec-49b6-8b16-f3170197c591","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"ac054af3-d589-4565-9644-e6af75975c53","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"3ec4f045-c714-4a22-bd1a-e07330b45ab3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"cddb9867-8e30-4951-9b13-3d1e78b9c45a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dc6d9911-785d-432f-8b53-56a1ccab302f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"77c88437-b587-496e-8b4d-1824ea8b5305","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ce963d96-5a8e-44c3-99de-c0dbebe234ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:50:46.724 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:46.742+0000 7f9cede25640 10 librados: watch_flush enter 2024-03-29T10:50:46.724 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:46.742+0000 7f9cede25640 10 librados: watch_flush exit 2024-03-29T10:50:46.725 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:46.742+0000 7f9cede25640 20 client.4631.objecter shutdown clearing up homeless session... 2024-03-29T10:50:46.725 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:46.742+0000 7f9cede25640 10 client.4631.objecter successfully canceled tick 2024-03-29T10:50:46.725 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:46.742+0000 7f9cede25640 1 -- 172.21.15.88:0/353643557 >> v1:172.21.15.88:6816/1752861349 conn(0x7f9cd406dcb0 legacy=0x7f9cd4070170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:46.725 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:46.742+0000 7f9cede25640 10 monclient: shutdown 2024-03-29T10:50:46.725 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:46.743+0000 7f9cede25640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:50:46.725 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:46.743+0000 7f9cede25640 1 -- 172.21.15.88:0/353643557 >> v1:172.21.15.88:6790/0 conn(0x7f9ce80b17d0 legacy=0x7f9ce812ed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:46.725 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:46.743+0000 7f9cede25640 1 -- 172.21.15.88:0/353643557 shutdown_connections 2024-03-29T10:50:46.726 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:46.743+0000 7f9cede25640 1 -- 172.21.15.88:0/353643557 >> 172.21.15.88:0/353643557 conn(0x7f9ce8066a00 msgr2=0x7f9ce80b2490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:50:46.726 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:46.743+0000 7f9cede25640 1 -- 172.21.15.88:0/353643557 shutdown_connections 2024-03-29T10:50:46.727 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:46.743+0000 7f9cede25640 1 -- 172.21.15.88:0/353643557 wait complete. 2024-03-29T10:50:46.727 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:46.744+0000 7f9cede25640 1 librados: shutdown 2024-03-29T10:50:46.743 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:50:53.550 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:50:53.703 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:53.722+0000 7fe241414640 10 monclient: get_monmap_and_config 2024-03-29T10:50:53.704 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:53.722+0000 7fe241414640 10 monclient: build_initial_monmap 2024-03-29T10:50:53.704 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:53.722+0000 7fe241414640 10 monclient: monmap: 2024-03-29T10:50:53.704 INFO:teuthology.misc.health.smithi088.stderr:epoch 0 2024-03-29T10:50:53.704 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:50:53.704 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:50:53.723082+0000 2024-03-29T10:50:53.704 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:50:53.723082+0000 2024-03-29T10:50:53.704 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 0 (unknown) 2024-03-29T10:50:53.704 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:50:53.704 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-03-29T10:50:53.704 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-29T10:50:53.704 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-b 2024-03-29T10:50:53.704 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:50:53.705 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:53.723+0000 7fe241414640 1 Processor -- start 2024-03-29T10:50:53.706 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:53.723+0000 7fe241414640 1 -- start start 2024-03-29T10:50:53.706 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:53.723+0000 7fe241414640 10 monclient: init 2024-03-29T10:50:53.706 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:53.724+0000 7fe241414640 10 monclient: _reopen_session rank -1 2024-03-29T10:50:53.706 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:53.724+0000 7fe241414640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:50:53.706 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:53.724+0000 7fe241414640 10 monclient(hunting): picked mon.noname-b con 0x7fe23c1420b0 addr v1:172.21.15.88:6790/0 2024-03-29T10:50:53.706 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:53.724+0000 7fe241414640 10 monclient(hunting): picked mon.noname-c con 0x7fe23c06d9d0 addr v1:172.21.15.196:6789/0 2024-03-29T10:50:53.706 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:53.724+0000 7fe241414640 10 monclient(hunting): picked mon.noname-a con 0x7fe23c0a9e70 addr v1:172.21.15.88:6789/0 2024-03-29T10:50:53.706 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:53.724+0000 7fe241414640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe23c13baf0 con 0x7fe23c0a9e70 2024-03-29T10:50:53.706 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:53.724+0000 7fe241414640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe23c0ab840 con 0x7fe23c06d9d0 2024-03-29T10:50:53.706 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:53.724+0000 7fe241414640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe23c0abab0 con 0x7fe23c1420b0 2024-03-29T10:50:53.706 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:53.724+0000 7fe241414640 10 monclient(hunting): _renew_subs 2024-03-29T10:50:53.706 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:53.724+0000 7fe241414640 10 monclient(hunting): authenticate will time out at 1375.960439s 2024-03-29T10:50:53.712 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:53.731+0000 7fe23bfff640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fe23c0a9e70 0x7fe23c0aa2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:35553/0 (socket says 172.21.15.88:35553) 2024-03-29T10:50:53.713 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:53.731+0000 7fe23bfff640 1 -- 172.21.15.88:0/2057774282 learned_addr learned my addr 172.21.15.88:0/2057774282 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-29T10:50:53.717 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:53.732+0000 7fe239ffb640 1 -- 172.21.15.88:0/2057774282 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2005829117 0 0) 0x7fe23c13baf0 con 0x7fe23c0a9e70 2024-03-29T10:50:53.717 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:53.732+0000 7fe239ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:50:53.717 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:53.732+0000 7fe239ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:50:53.717 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:53.732+0000 7fe239ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:50:53.717 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:53.732+0000 7fe239ffb640 10 monclient(hunting): my global_id is 4590 2024-03-29T10:50:53.717 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:53.732+0000 7fe239ffb640 1 -- 172.21.15.88:0/2057774282 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe224003520 con 0x7fe23c0a9e70 2024-03-29T10:50:53.717 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:53.732+0000 7fe239ffb640 1 -- 172.21.15.88:0/2057774282 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1849332324 0 0) 0x7fe23c0abab0 con 0x7fe23c1420b0 2024-03-29T10:50:53.717 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:53.732+0000 7fe239ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:50:53.717 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:53.732+0000 7fe239ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:50:53.718 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:53.732+0000 7fe239ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:50:53.718 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:53.732+0000 7fe239ffb640 10 monclient(hunting): my global_id is 4634 2024-03-29T10:50:53.718 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:53.732+0000 7fe239ffb640 1 -- 172.21.15.88:0/2057774282 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe23c13baf0 con 0x7fe23c1420b0 2024-03-29T10:50:53.718 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:53.732+0000 7fe239ffb640 1 -- 172.21.15.88:0/2057774282 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2443828633 0 0) 0x7fe224003520 con 0x7fe23c0a9e70 2024-03-29T10:50:53.718 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:53.732+0000 7fe239ffb640 1 -- 172.21.15.88:0/2057774282 >> v1:172.21.15.88:6790/0 conn(0x7fe23c1420b0 legacy=0x7fe23c0af1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:53.718 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:53.733+0000 7fe239ffb640 1 -- 172.21.15.88:0/2057774282 >> v1:172.21.15.196:6789/0 conn(0x7fe23c06d9d0 legacy=0x7fe23c0ae0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:50:53.718 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:53.733+0000 7fe239ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:50:53.718 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:53.733+0000 7fe239ffb640 1 monclient: found mon.noname-a 2024-03-29T10:50:53.718 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:53.733+0000 7fe239ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:50:53.718 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:53.733+0000 7fe239ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.88:6789/0 2024-03-29T10:50:53.718 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:53.733+0000 7fe239ffb640 1 -- 172.21.15.88:0/2057774282 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe23c141b90 con 0x7fe23c0a9e70 2024-03-29T10:50:53.718 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:53.733+0000 7fe239ffb640 10 monclient: _finish_auth 0 2024-03-29T10:50:53.718 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:53.733+0000 7fe239ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:50:53.719 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:53.733+0000 7fe239ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:50:53.719 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:53.733+0000 7fe241414640 5 monclient: authenticate success, global_id 4590 2024-03-29T10:50:53.719 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:53.733+0000 7fe241414640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:50:53.719 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:53.733+0000 7fe239ffb640 1 -- 172.21.15.88:0/2057774282 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7fe22c001690 con 0x7fe23c0a9e70 2024-03-29T10:50:53.719 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:53.733+0000 7fe239ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:50:53.719 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:53.733+0000 7fe239ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:50:53.719 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:53.733+0000 7fe239ffb640 10 monclient: dump: 2024-03-29T10:50:53.719 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:50:53.719 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:50:53.719 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:50:53.719 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:50:53.719 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:50:53.719 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:50:53.719 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:50:53.719 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:50:53.720 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:50:53.720 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:50:53.720 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:53.733+0000 7fe241414640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:50:53.720 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:53.733+0000 7fe239ffb640 1 -- 172.21.15.88:0/2057774282 <== mon.0 v1:172.21.15.88:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe22c0018b0 con 0x7fe23c0a9e70 2024-03-29T10:50:53.720 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:53.733+0000 7fe239ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:50:53.720 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:53.733+0000 7fe241414640 10 monclient: get_monmap_and_config success 2024-03-29T10:50:53.720 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:53.733+0000 7fe241414640 4 set_mon_vals no callback set 2024-03-29T10:50:53.720 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:53.734+0000 7fe239ffb640 1 -- 172.21.15.88:0/2057774282 <== mon.0 v1:172.21.15.88:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7fe22c001c40 con 0x7fe23c0a9e70 2024-03-29T10:50:53.720 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:53.734+0000 7fe239ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:50:53.720 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:53.734+0000 7fe239ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:50:53.720 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:53.734+0000 7fe239ffb640 10 monclient: dump: 2024-03-29T10:50:53.720 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:50:53.720 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:50:53.720 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:50:53.721 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:50:53.721 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:50:53.721 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:50:53.721 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:50:53.721 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:50:53.721 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:50:53.721 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:50:53.721 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:53.734+0000 7fe241414640 10 monclient: shutdown 2024-03-29T10:50:53.721 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:53.734+0000 7fe241414640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:50:53.721 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:53.734+0000 7fe241414640 1 -- 172.21.15.88:0/2057774282 >> v1:172.21.15.88:6789/0 conn(0x7fe23c0a9e70 legacy=0x7fe23c0aa2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:53.721 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:53.735+0000 7fe241414640 1 -- 172.21.15.88:0/2057774282 shutdown_connections 2024-03-29T10:50:53.721 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:53.735+0000 7fe241414640 1 -- 172.21.15.88:0/2057774282 >> 172.21.15.88:0/2057774282 conn(0x7fe23c0b36e0 msgr2=0x7fe23c0b3af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:50:53.722 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:53.735+0000 7fe241414640 1 -- 172.21.15.88:0/2057774282 shutdown_connections 2024-03-29T10:50:53.722 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:53.735+0000 7fe241414640 1 -- 172.21.15.88:0/2057774282 wait complete. 2024-03-29T10:50:53.722 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:53.736+0000 7fe241414640 10 monclient: build_initial_monmap 2024-03-29T10:50:53.722 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:53.736+0000 7fe241414640 10 monclient: monmap: 2024-03-29T10:50:53.722 INFO:teuthology.misc.health.smithi088.stderr:epoch 0 2024-03-29T10:50:53.722 INFO:teuthology.misc.health.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:50:53.722 INFO:teuthology.misc.health.smithi088.stderr:last_changed 0.000000 2024-03-29T10:50:53.722 INFO:teuthology.misc.health.smithi088.stderr:created 0.000000 2024-03-29T10:50:53.722 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 0 (unknown) 2024-03-29T10:50:53.722 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:50:53.722 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-03-29T10:50:53.723 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-29T10:50:53.723 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-c 2024-03-29T10:50:53.723 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:50:53.724 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:53.736+0000 7fe241414640 1 librados: starting msgr at 2024-03-29T10:50:53.725 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:53.736+0000 7fe241414640 1 librados: starting objecter 2024-03-29T10:50:53.725 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:53.737+0000 7fe241414640 1 Processor -- start 2024-03-29T10:50:53.725 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:53.737+0000 7fe241414640 1 -- start start 2024-03-29T10:50:53.725 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:53.737+0000 7fe241414640 1 librados: setting wanted keys 2024-03-29T10:50:53.725 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:53.737+0000 7fe241414640 1 librados: calling monclient init 2024-03-29T10:50:53.725 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:53.737+0000 7fe241414640 10 monclient: init 2024-03-29T10:50:53.725 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:53.737+0000 7fe241414640 10 monclient: _reopen_session rank -1 2024-03-29T10:50:53.725 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:53.737+0000 7fe241414640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:50:53.725 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:53.737+0000 7fe241414640 10 monclient(hunting): picked mon.noname-c con 0x7fe23c06d9d0 addr v1:172.21.15.88:6790/0 2024-03-29T10:50:53.725 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:53.737+0000 7fe241414640 10 monclient(hunting): picked mon.noname-a con 0x7fe23c0a9e70 addr v1:172.21.15.88:6789/0 2024-03-29T10:50:53.725 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:53.737+0000 7fe241414640 10 monclient(hunting): picked mon.noname-b con 0x7fe23c1420b0 addr v1:172.21.15.196:6789/0 2024-03-29T10:50:53.725 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:53.737+0000 7fe241414640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe23c144460 con 0x7fe23c0a9e70 2024-03-29T10:50:53.725 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:53.737+0000 7fe241414640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe23c147c30 con 0x7fe23c1420b0 2024-03-29T10:50:53.726 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:53.737+0000 7fe241414640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe23c147f10 con 0x7fe23c06d9d0 2024-03-29T10:50:53.726 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:53.737+0000 7fe241414640 10 monclient(hunting): _renew_subs 2024-03-29T10:50:53.726 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:53.737+0000 7fe241414640 10 monclient(hunting): authenticate will time out at 1375.973886s 2024-03-29T10:50:53.726 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:53.741+0000 7fe23affd640 1 --1- >> v1:172.21.15.88:6790/0 conn(0x7fe23c06d9d0 0x7fe23c1deb50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6790/0 says I am v1:172.21.15.88:45819/0 (socket says 172.21.15.88:45819) 2024-03-29T10:50:53.726 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:53.741+0000 7fe23affd640 1 -- 172.21.15.88:0/141789571 learned_addr learned my addr 172.21.15.88:0/141789571 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-29T10:50:53.726 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:53.741+0000 7fe2237fe640 10 client.?.objecter ms_handle_connect 0x7fe23c06d9d0 2024-03-29T10:50:53.726 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:53.741+0000 7fe2237fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:50:53.726 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:53.741+0000 7fe2237fe640 1 -- 172.21.15.88:0/141789571 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3113187626 0 0) 0x7fe23c147f10 con 0x7fe23c06d9d0 2024-03-29T10:50:53.726 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:53.741+0000 7fe2237fe640 10 monclient(hunting): _negotiate 2024-03-29T10:50:53.726 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:53.741+0000 7fe2237fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:50:53.726 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:53.741+0000 7fe2237fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:50:53.726 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:53.741+0000 7fe2237fe640 10 monclient(hunting): my global_id is 4637 2024-03-29T10:50:53.726 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:53.741+0000 7fe2237fe640 1 -- 172.21.15.88:0/141789571 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe228004eb0 con 0x7fe23c06d9d0 2024-03-29T10:50:53.727 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:53.742+0000 7fe2237fe640 10 client.?.objecter ms_handle_connect 0x7fe23c0a9e70 2024-03-29T10:50:53.727 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:53.742+0000 7fe2237fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:50:53.727 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:53.742+0000 7fe2237fe640 1 -- 172.21.15.88:0/141789571 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2323400417 0 0) 0x7fe228004eb0 con 0x7fe23c06d9d0 2024-03-29T10:50:53.727 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:53.742+0000 7fe2237fe640 1 -- 172.21.15.88:0/141789571 --> v1:172.21.15.88:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe23c147f10 con 0x7fe23c06d9d0 2024-03-29T10:50:53.727 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:53.742+0000 7fe2237fe640 1 -- 172.21.15.88:0/141789571 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7fe234001d20 con 0x7fe23c06d9d0 2024-03-29T10:50:53.727 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:53.742+0000 7fe2237fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:50:53.727 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:53.742+0000 7fe2237fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:50:53.727 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:53.742+0000 7fe2237fe640 10 monclient(hunting): dump: 2024-03-29T10:50:53.727 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:50:53.727 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:50:53.727 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:50:53.727 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:50:53.727 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:50:53.727 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:50:53.727 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:50:53.728 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:50:53.728 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:50:53.728 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:50:53.728 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:53.742+0000 7fe2237fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:50:53.728 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:53.743+0000 7fe2237fe640 1 -- 172.21.15.88:0/141789571 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 392628435 0 0) 0x7fe23c144460 con 0x7fe23c0a9e70 2024-03-29T10:50:53.728 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:53.743+0000 7fe2237fe640 10 monclient(hunting): _negotiate 2024-03-29T10:50:53.728 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:53.743+0000 7fe2237fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:50:53.728 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:53.743+0000 7fe2237fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:50:53.728 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:53.743+0000 7fe2237fe640 10 monclient(hunting): my global_id is 4593 2024-03-29T10:50:53.728 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:53.743+0000 7fe2237fe640 1 -- 172.21.15.88:0/141789571 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe228004eb0 con 0x7fe23c0a9e70 2024-03-29T10:50:53.728 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:53.743+0000 7fe2237fe640 1 -- 172.21.15.88:0/141789571 <== mon.2 v1:172.21.15.88:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3502283736 0 0) 0x7fe23c147f10 con 0x7fe23c06d9d0 2024-03-29T10:50:53.728 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:53.743+0000 7fe2237fe640 1 -- 172.21.15.88:0/141789571 >> v1:172.21.15.196:6789/0 conn(0x7fe23c1420b0 legacy=0x7fe23c1e6e90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:50:53.729 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:53.745+0000 7fe2237fe640 1 -- 172.21.15.88:0/141789571 >> v1:172.21.15.88:6789/0 conn(0x7fe23c0a9e70 legacy=0x7fe23c1dbb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:53.729 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:53.745+0000 7fe2237fe640 10 monclient: _finish_hunting 0 2024-03-29T10:50:53.729 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:53.745+0000 7fe2237fe640 1 monclient: found mon.c 2024-03-29T10:50:53.729 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:53.745+0000 7fe2237fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:50:53.729 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:53.745+0000 7fe2237fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-03-29T10:50:53.729 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:53.745+0000 7fe2237fe640 1 -- 172.21.15.88:0/141789571 --> v1:172.21.15.88:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe23c1481f0 con 0x7fe23c06d9d0 2024-03-29T10:50:53.729 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:53.745+0000 7fe2237fe640 10 monclient: _finish_auth 0 2024-03-29T10:50:53.729 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:53.745+0000 7fe2237fe640 10 monclient: _check_auth_tickets 2024-03-29T10:50:53.729 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:53.745+0000 7fe2237fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:50:53.729 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:53.745+0000 7fe241414640 5 monclient: authenticate success, global_id 4637 2024-03-29T10:50:53.729 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:53.745+0000 7fe241414640 10 monclient: _renew_subs 2024-03-29T10:50:53.730 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:53.745+0000 7fe241414640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-03-29T10:50:53.730 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:53.745+0000 7fe241414640 1 -- 172.21.15.88:0/141789571 --> v1:172.21.15.88:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe23c1413e0 con 0x7fe23c06d9d0 2024-03-29T10:50:53.730 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:53.745+0000 7fe241414640 10 client.4637.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:50:53.730 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:53.745+0000 7fe241414640 10 monclient: _renew_subs 2024-03-29T10:50:53.730 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:53.745+0000 7fe241414640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-03-29T10:50:53.730 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:53.745+0000 7fe241414640 1 -- 172.21.15.88:0/141789571 --> v1:172.21.15.88:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe23c1478c0 con 0x7fe23c06d9d0 2024-03-29T10:50:53.730 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:53.745+0000 7fe241414640 1 librados: init done 2024-03-29T10:50:53.731 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:53.748+0000 7fe241414640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:50:53.731 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:53.748+0000 7fe241414640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:50:53.731 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:53.748+0000 7fe241414640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-03-29T10:50:53.731 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:53.748+0000 7fe241414640 1 -- 172.21.15.88:0/141789571 --> v1:172.21.15.88:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe23c0008d0 con 0x7fe23c06d9d0 2024-03-29T10:50:53.731 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:53.749+0000 7fe2237fe640 1 -- 172.21.15.88:0/141789571 <== mon.2 v1:172.21.15.88:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe234002000 con 0x7fe23c06d9d0 2024-03-29T10:50:53.732 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:53.749+0000 7fe2237fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:50:53.732 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:53.749+0000 7fe2397fa640 4 set_mon_vals no callback set 2024-03-29T10:50:53.732 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:53.750+0000 7fe2237fe640 1 -- 172.21.15.88:0/141789571 <== mon.2 v1:172.21.15.88:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7fe234003e10 con 0x7fe23c06d9d0 2024-03-29T10:50:53.732 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:53.750+0000 7fe2237fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:50:53.732 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:53.750+0000 7fe2237fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:50:53.732 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:53.750+0000 7fe2237fe640 10 monclient: dump: 2024-03-29T10:50:53.732 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:50:53.732 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:50:53.732 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:50:53.732 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:50:53.732 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:50:53.732 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:50:53.732 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:50:53.732 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:50:53.733 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:50:53.733 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:50:53.733 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:53.750+0000 7fe2237fe640 1 -- 172.21.15.88:0/141789571 <== mon.2 v1:172.21.15.88:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4235629220 0 0) 0x7fe234004030 con 0x7fe23c06d9d0 2024-03-29T10:50:53.734 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:53.751+0000 7fe2237fe640 1 -- 172.21.15.88:0/141789571 <== mon.2 v1:172.21.15.88:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1407902845 0 0) 0x7fe234084df0 con 0x7fe23c06d9d0 2024-03-29T10:50:53.734 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:53.751+0000 7fe2237fe640 10 client.4637.objecter ms_dispatch 0x7fe23c000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T10:50:53.734 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:53.751+0000 7fe2237fe640 3 client.4637.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T10:50:53.734 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:53.751+0000 7fe2237fe640 3 client.4637.objecter handle_osd_map decoding full epoch 11 2024-03-29T10:50:53.734 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:53.751+0000 7fe2237fe640 20 client.4637.objecter dump_active .. 0 homeless 2024-03-29T10:50:53.734 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:53.751+0000 7fe23a7fc640 10 monclient: get_auth_request con 0x7fe22806dce0 auth_method 0 2024-03-29T10:50:53.734 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:53.751+0000 7fe2237fe640 10 client.4637.objecter ms_handle_connect 0x7fe22806dce0 2024-03-29T10:50:53.742 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:53.758+0000 7fe2237fe640 1 -- 172.21.15.88:0/141789571 <== mon.2 v1:172.21.15.88:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe234050790 con 0x7fe23c06d9d0 2024-03-29T10:50:53.742 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:53.758+0000 7fe2237fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:50:53.742 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:53.758+0000 7fe2237fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:50:54.059 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:54.077+0000 7fe241414640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:50:54.059 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:54.077+0000 7fe241414640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:50:54.059 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:54.077+0000 7fe241414640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-03-29T10:50:54.059 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:54.077+0000 7fe241414640 1 -- 172.21.15.88:0/141789571 --> v1:172.21.15.88:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe23c144180 con 0x7fe23c06d9d0 2024-03-29T10:50:54.061 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:54.079+0000 7fe2237fe640 1 -- 172.21.15.88:0/141789571 <== mon.2 v1:172.21.15.88:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 1123663608) 0x7fe23c144180 con 0x7fe23c06d9d0 2024-03-29T10:50:54.061 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:54.079+0000 7fe2237fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:50:54.061 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:54.079+0000 7fe2237fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:50:54.061 INFO:teuthology.misc.health.smithi088.stdout: 2024-03-29T10:50:54.061 INFO:teuthology.misc.health.smithi088.stdout:{"epoch":11,"fsid":"869f5188-f92b-42dd-a46d-8c646e0388a9","created":"2024-03-29T10:40:40.285087+0000","modified":"2024-03-29T10:50:45.236723+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:50:45.236723+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a26c0c1c-8de0-438c-a3f6-3df10ea61574","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"5de24915-8dec-49b6-8b16-f3170197c591","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"ac054af3-d589-4565-9644-e6af75975c53","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"3ec4f045-c714-4a22-bd1a-e07330b45ab3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"cddb9867-8e30-4951-9b13-3d1e78b9c45a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dc6d9911-785d-432f-8b53-56a1ccab302f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"77c88437-b587-496e-8b4d-1824ea8b5305","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ce963d96-5a8e-44c3-99de-c0dbebe234ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:50:54.067 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:54.082+0000 7fe241414640 10 librados: watch_flush enter 2024-03-29T10:50:54.067 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:54.083+0000 7fe241414640 10 librados: watch_flush exit 2024-03-29T10:50:54.067 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:54.083+0000 7fe241414640 20 client.4637.objecter shutdown clearing up homeless session... 2024-03-29T10:50:54.067 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:54.083+0000 7fe241414640 10 client.4637.objecter successfully canceled tick 2024-03-29T10:50:54.067 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:54.083+0000 7fe241414640 1 -- 172.21.15.88:0/141789571 >> v1:172.21.15.88:6816/1752861349 conn(0x7fe22806dce0 legacy=0x7fe2280701a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:54.067 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:54.083+0000 7fe241414640 10 monclient: shutdown 2024-03-29T10:50:54.067 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:54.083+0000 7fe241414640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:50:54.067 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:54.084+0000 7fe241414640 1 -- 172.21.15.88:0/141789571 >> v1:172.21.15.88:6790/0 conn(0x7fe23c06d9d0 legacy=0x7fe23c1deb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:54.068 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:54.084+0000 7fe241414640 1 -- 172.21.15.88:0/141789571 shutdown_connections 2024-03-29T10:50:54.068 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:54.084+0000 7fe241414640 1 -- 172.21.15.88:0/141789571 >> 172.21.15.88:0/141789571 conn(0x7fe23c0b36e0 msgr2=0x7fe23c0adfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:50:54.068 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:54.085+0000 7fe241414640 1 -- 172.21.15.88:0/141789571 shutdown_connections 2024-03-29T10:50:54.068 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:54.085+0000 7fe241414640 1 -- 172.21.15.88:0/141789571 wait complete. 2024-03-29T10:50:54.068 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:50:54.086+0000 7fe241414640 1 librados: shutdown 2024-03-29T10:50:54.086 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:51:00.892 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:51:01.053 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:01.071+0000 7f9ae1c77640 10 monclient: get_monmap_and_config 2024-03-29T10:51:01.053 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:01.071+0000 7f9ae1c77640 10 monclient: build_initial_monmap 2024-03-29T10:51:01.053 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:01.071+0000 7f9ae1c77640 10 monclient: monmap: 2024-03-29T10:51:01.053 INFO:teuthology.misc.health.smithi088.stderr:epoch 0 2024-03-29T10:51:01.053 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:51:01.053 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:51:01.072385+0000 2024-03-29T10:51:01.053 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:51:01.072385+0000 2024-03-29T10:51:01.053 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 0 (unknown) 2024-03-29T10:51:01.054 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:51:01.054 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-03-29T10:51:01.054 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-29T10:51:01.054 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-b 2024-03-29T10:51:01.054 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:51:01.054 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:01.072+0000 7f9ae1c77640 1 Processor -- start 2024-03-29T10:51:01.055 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:01.072+0000 7f9ae1c77640 1 -- start start 2024-03-29T10:51:01.055 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:01.072+0000 7f9ae1c77640 10 monclient: init 2024-03-29T10:51:01.056 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:01.073+0000 7f9ae1c77640 10 monclient: _reopen_session rank -1 2024-03-29T10:51:01.056 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:01.073+0000 7f9ae1c77640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:51:01.056 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:01.073+0000 7f9ae1c77640 10 monclient(hunting): picked mon.noname-a con 0x7f9adc0b2800 addr v1:172.21.15.88:6789/0 2024-03-29T10:51:01.056 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:01.073+0000 7f9ae1c77640 10 monclient(hunting): picked mon.noname-c con 0x7f9adc142b20 addr v1:172.21.15.196:6789/0 2024-03-29T10:51:01.056 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:01.073+0000 7f9ae1c77640 10 monclient(hunting): picked mon.noname-b con 0x7f9adc13c040 addr v1:172.21.15.88:6790/0 2024-03-29T10:51:01.056 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:01.073+0000 7f9ae1c77640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9adc0aad10 con 0x7f9adc0b2800 2024-03-29T10:51:01.056 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:01.073+0000 7f9ae1c77640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9adc141f30 con 0x7f9adc142b20 2024-03-29T10:51:01.056 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:01.073+0000 7f9ae1c77640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9adc0aaf60 con 0x7f9adc13c040 2024-03-29T10:51:01.056 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:01.073+0000 7f9ae1c77640 10 monclient(hunting): _renew_subs 2024-03-29T10:51:01.057 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:01.073+0000 7f9ae1c77640 10 monclient(hunting): authenticate will time out at 1383.309861s 2024-03-29T10:51:01.058 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:01.076+0000 7f9ae0c75640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f9adc0b2800 0x7f9adc142410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:60452/0 (socket says 172.21.15.88:60452) 2024-03-29T10:51:01.058 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:01.076+0000 7f9ae0c75640 1 -- 172.21.15.88:0/2590251761 learned_addr learned my addr 172.21.15.88:0/2590251761 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-29T10:51:01.060 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:01.078+0000 7f9adb7fe640 1 -- 172.21.15.88:0/2590251761 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4166458133 0 0) 0x7f9adc0aad10 con 0x7f9adc0b2800 2024-03-29T10:51:01.061 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:01.078+0000 7f9adb7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:51:01.061 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:01.078+0000 7f9adb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:01.061 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:01.078+0000 7f9adb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:01.061 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:01.078+0000 7f9adb7fe640 10 monclient(hunting): my global_id is 4596 2024-03-29T10:51:01.061 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:01.078+0000 7f9adb7fe640 1 -- 172.21.15.88:0/2590251761 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9ac4003520 con 0x7f9adc0b2800 2024-03-29T10:51:01.061 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:01.078+0000 7f9adb7fe640 1 -- 172.21.15.88:0/2590251761 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4288129847 0 0) 0x7f9adc0aaf60 con 0x7f9adc13c040 2024-03-29T10:51:01.061 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:01.078+0000 7f9adb7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:51:01.061 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:01.078+0000 7f9adb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:01.061 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:01.078+0000 7f9adb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:01.061 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:01.078+0000 7f9adb7fe640 10 monclient(hunting): my global_id is 4640 2024-03-29T10:51:01.061 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:01.078+0000 7f9adb7fe640 1 -- 172.21.15.88:0/2590251761 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9adc0aad10 con 0x7f9adc13c040 2024-03-29T10:51:01.061 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:01.079+0000 7f9adb7fe640 1 -- 172.21.15.88:0/2590251761 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3837190797 0 0) 0x7f9adc141f30 con 0x7f9adc142b20 2024-03-29T10:51:01.061 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:01.079+0000 7f9adb7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:51:01.061 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:01.079+0000 7f9adb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:01.062 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:01.079+0000 7f9adb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:01.062 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:01.079+0000 7f9adb7fe640 10 monclient(hunting): my global_id is 4570 2024-03-29T10:51:01.062 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:01.079+0000 7f9adb7fe640 1 -- 172.21.15.88:0/2590251761 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9adc0aaf60 con 0x7f9adc142b20 2024-03-29T10:51:01.062 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:01.079+0000 7f9adb7fe640 1 -- 172.21.15.88:0/2590251761 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3549550317 0 0) 0x7f9ac4003520 con 0x7f9adc0b2800 2024-03-29T10:51:01.062 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:01.079+0000 7f9adb7fe640 1 -- 172.21.15.88:0/2590251761 >> v1:172.21.15.88:6790/0 conn(0x7f9adc13c040 legacy=0x7f9adc14d7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:01.062 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:01.079+0000 7f9adb7fe640 1 -- 172.21.15.88:0/2590251761 >> v1:172.21.15.196:6789/0 conn(0x7f9adc142b20 legacy=0x7f9adc13b930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:01.062 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:01.079+0000 7f9adb7fe640 10 monclient: _finish_hunting 0 2024-03-29T10:51:01.062 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:01.079+0000 7f9adb7fe640 1 monclient: found mon.noname-a 2024-03-29T10:51:01.062 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:01.079+0000 7f9adb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:51:01.062 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:01.079+0000 7f9adb7fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.88:6789/0 2024-03-29T10:51:01.062 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:01.079+0000 7f9adb7fe640 1 -- 172.21.15.88:0/2590251761 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9adc0b3900 con 0x7f9adc0b2800 2024-03-29T10:51:01.062 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:01.079+0000 7f9adb7fe640 10 monclient: _finish_auth 0 2024-03-29T10:51:01.062 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:01.079+0000 7f9adb7fe640 10 monclient: _check_auth_tickets 2024-03-29T10:51:01.062 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:01.079+0000 7f9adb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:51:01.063 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:01.079+0000 7f9adb7fe640 1 -- 172.21.15.88:0/2590251761 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f9ad4001a10 con 0x7f9adc0b2800 2024-03-29T10:51:01.063 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:01.079+0000 7f9adb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:01.063 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:01.079+0000 7f9adb7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:51:01.063 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:01.079+0000 7f9adb7fe640 10 monclient: dump: 2024-03-29T10:51:01.063 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:51:01.063 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:51:01.063 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:51:01.063 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:51:01.063 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:01.063 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:51:01.063 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:51:01.063 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:51:01.063 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:51:01.063 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:51:01.064 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:01.080+0000 7f9ae1c77640 5 monclient: authenticate success, global_id 4596 2024-03-29T10:51:01.064 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:01.080+0000 7f9ae1c77640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:51:01.065 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:01.083+0000 7f9adb7fe640 1 -- 172.21.15.88:0/2590251761 <== mon.0 v1:172.21.15.88:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9ad4001b70 con 0x7f9adc0b2800 2024-03-29T10:51:01.065 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:01.083+0000 7f9adb7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:51:01.065 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:01.083+0000 7f9adb7fe640 1 -- 172.21.15.88:0/2590251761 <== mon.0 v1:172.21.15.88:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f9ad4001fc0 con 0x7f9adc0b2800 2024-03-29T10:51:01.065 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:01.083+0000 7f9adb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:01.065 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:01.083+0000 7f9adb7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:51:01.065 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:01.083+0000 7f9adb7fe640 10 monclient: dump: 2024-03-29T10:51:01.066 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:51:01.066 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:51:01.066 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:51:01.066 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:51:01.066 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:01.066 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:51:01.066 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:51:01.066 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:51:01.066 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:51:01.066 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:51:01.066 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:01.083+0000 7f9ae1c77640 10 monclient: get_monmap_and_config success 2024-03-29T10:51:01.066 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:01.083+0000 7f9ae1c77640 4 set_mon_vals no callback set 2024-03-29T10:51:01.067 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:01.084+0000 7f9ae1c77640 10 monclient: shutdown 2024-03-29T10:51:01.067 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:01.084+0000 7f9ae1c77640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:51:01.067 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:01.084+0000 7f9ae1c77640 1 -- 172.21.15.88:0/2590251761 >> v1:172.21.15.88:6789/0 conn(0x7f9adc0b2800 legacy=0x7f9adc142410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:01.067 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:01.084+0000 7f9ae1c77640 1 -- 172.21.15.88:0/2590251761 shutdown_connections 2024-03-29T10:51:01.067 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:01.084+0000 7f9ae1c77640 1 -- 172.21.15.88:0/2590251761 >> 172.21.15.88:0/2590251761 conn(0x7f9adc06d0e0 msgr2=0x7f9adc066a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:51:01.068 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:01.085+0000 7f9ae1c77640 1 -- 172.21.15.88:0/2590251761 shutdown_connections 2024-03-29T10:51:01.069 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:01.086+0000 7f9ae1c77640 1 -- 172.21.15.88:0/2590251761 wait complete. 2024-03-29T10:51:01.069 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:01.087+0000 7f9ae1c77640 10 monclient: build_initial_monmap 2024-03-29T10:51:01.069 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:01.087+0000 7f9ae1c77640 10 monclient: monmap: 2024-03-29T10:51:01.069 INFO:teuthology.misc.health.smithi088.stderr:epoch 0 2024-03-29T10:51:01.069 INFO:teuthology.misc.health.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:51:01.069 INFO:teuthology.misc.health.smithi088.stderr:last_changed 0.000000 2024-03-29T10:51:01.069 INFO:teuthology.misc.health.smithi088.stderr:created 0.000000 2024-03-29T10:51:01.069 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 0 (unknown) 2024-03-29T10:51:01.069 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:51:01.069 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-03-29T10:51:01.069 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-29T10:51:01.069 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-c 2024-03-29T10:51:01.069 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:51:01.070 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:01.088+0000 7f9ae1c77640 1 librados: starting msgr at 2024-03-29T10:51:01.070 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:01.088+0000 7f9ae1c77640 1 librados: starting objecter 2024-03-29T10:51:01.070 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:01.088+0000 7f9ae1c77640 1 Processor -- start 2024-03-29T10:51:01.071 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:01.088+0000 7f9ae1c77640 1 -- start start 2024-03-29T10:51:01.071 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:01.088+0000 7f9ae1c77640 1 librados: setting wanted keys 2024-03-29T10:51:01.071 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:01.088+0000 7f9ae1c77640 1 librados: calling monclient init 2024-03-29T10:51:01.071 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:01.088+0000 7f9ae1c77640 10 monclient: init 2024-03-29T10:51:01.071 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:01.089+0000 7f9ae1c77640 10 monclient: _reopen_session rank -1 2024-03-29T10:51:01.071 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:01.090+0000 7f9ae1c77640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:51:01.072 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:01.090+0000 7f9ae1c77640 10 monclient(hunting): picked mon.noname-a con 0x7f9adc0b2800 addr v1:172.21.15.88:6789/0 2024-03-29T10:51:01.072 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:01.090+0000 7f9ae0c75640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f9adc0b2800 0x7f9adc1e7130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:34064/0 (socket says 172.21.15.88:34064) 2024-03-29T10:51:01.072 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:01.090+0000 7f9ae0c75640 1 -- 172.21.15.88:0/316581927 learned_addr learned my addr 172.21.15.88:0/316581927 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-29T10:51:01.075 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:01.093+0000 7f9ad97fa640 10 client.?.objecter ms_handle_connect 0x7f9adc0b2800 2024-03-29T10:51:01.075 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:01.093+0000 7f9ad97fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:51:01.075 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:01.093+0000 7f9ae1c77640 10 monclient(hunting): picked mon.noname-c con 0x7f9adc13c040 addr v1:172.21.15.88:6790/0 2024-03-29T10:51:01.075 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:01.093+0000 7f9ae1c77640 10 monclient(hunting): picked mon.noname-b con 0x7f9adc142b20 addr v1:172.21.15.196:6789/0 2024-03-29T10:51:01.075 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:01.093+0000 7f9ae1c77640 1 -- 172.21.15.88:0/316581927 --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9adc13e980 con 0x7f9adc0b2800 2024-03-29T10:51:01.075 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:01.093+0000 7f9ae1c77640 1 -- 172.21.15.88:0/316581927 --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9adc13d850 con 0x7f9adc142b20 2024-03-29T10:51:01.075 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:01.093+0000 7f9ae1c77640 1 -- 172.21.15.88:0/316581927 --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9adc13d9e0 con 0x7f9adc13c040 2024-03-29T10:51:01.075 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:01.093+0000 7f9ae1c77640 10 monclient(hunting): _renew_subs 2024-03-29T10:51:01.075 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:01.093+0000 7f9ae1c77640 10 monclient(hunting): authenticate will time out at 1383.329425s 2024-03-29T10:51:01.077 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:01.094+0000 7f9ad97fa640 10 client.?.objecter ms_handle_connect 0x7f9adc13c040 2024-03-29T10:51:01.077 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:01.095+0000 7f9ad97fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:51:01.077 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:01.095+0000 7f9ad97fa640 1 -- 172.21.15.88:0/316581927 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 360869520 0 0) 0x7f9adc13d9e0 con 0x7f9adc13c040 2024-03-29T10:51:01.077 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:01.095+0000 7f9ad97fa640 10 monclient(hunting): _negotiate 2024-03-29T10:51:01.077 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:01.095+0000 7f9ad97fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:01.077 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:01.095+0000 7f9ad97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:01.077 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:01.095+0000 7f9ad97fa640 10 monclient(hunting): my global_id is 4643 2024-03-29T10:51:01.077 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:01.095+0000 7f9ad97fa640 1 -- 172.21.15.88:0/316581927 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9ac8004eb0 con 0x7f9adc13c040 2024-03-29T10:51:01.078 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:01.096+0000 7f9ad97fa640 1 -- 172.21.15.88:0/316581927 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2330010181 0 0) 0x7f9ac8004eb0 con 0x7f9adc13c040 2024-03-29T10:51:01.079 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:01.096+0000 7f9ad97fa640 1 -- 172.21.15.88:0/316581927 --> v1:172.21.15.88:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9adc13d9e0 con 0x7f9adc13c040 2024-03-29T10:51:01.079 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:01.096+0000 7f9ad97fa640 1 -- 172.21.15.88:0/316581927 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f9acc001820 con 0x7f9adc13c040 2024-03-29T10:51:01.079 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:01.097+0000 7f9ad97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:01.079 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:01.097+0000 7f9ad97fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:51:01.079 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:01.097+0000 7f9ad97fa640 10 monclient(hunting): dump: 2024-03-29T10:51:01.079 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:51:01.079 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:51:01.079 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:51:01.079 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:51:01.079 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:01.079 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:51:01.079 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:51:01.079 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:51:01.080 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:51:01.080 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:51:01.080 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:01.097+0000 7f9ad97fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:51:01.080 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:01.097+0000 7f9ad97fa640 1 -- 172.21.15.88:0/316581927 <== mon.2 v1:172.21.15.88:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2734221974 0 0) 0x7f9adc13d9e0 con 0x7f9adc13c040 2024-03-29T10:51:01.080 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:01.097+0000 7f9ad97fa640 1 -- 172.21.15.88:0/316581927 >> v1:172.21.15.196:6789/0 conn(0x7f9adc142b20 legacy=0x7f9adc1ee6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:01.080 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:01.097+0000 7f9ad97fa640 1 -- 172.21.15.88:0/316581927 >> v1:172.21.15.88:6789/0 conn(0x7f9adc0b2800 legacy=0x7f9adc1e7130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:01.080 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:01.097+0000 7f9ad97fa640 10 monclient: _finish_hunting 0 2024-03-29T10:51:01.080 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:01.097+0000 7f9ad97fa640 1 monclient: found mon.c 2024-03-29T10:51:01.080 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:01.097+0000 7f9ad97fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:51:01.080 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:01.097+0000 7f9ad97fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-03-29T10:51:01.080 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:01.097+0000 7f9ad97fa640 1 -- 172.21.15.88:0/316581927 --> v1:172.21.15.88:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9adc138f50 con 0x7f9adc13c040 2024-03-29T10:51:01.080 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:01.097+0000 7f9ad97fa640 10 monclient: _finish_auth 0 2024-03-29T10:51:01.080 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:01.097+0000 7f9ad97fa640 10 monclient: _check_auth_tickets 2024-03-29T10:51:01.080 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:01.097+0000 7f9ad97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:51:01.080 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:01.097+0000 7f9ae1c77640 5 monclient: authenticate success, global_id 4643 2024-03-29T10:51:01.081 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:01.097+0000 7f9ae1c77640 10 monclient: _renew_subs 2024-03-29T10:51:01.081 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:01.097+0000 7f9ae1c77640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-03-29T10:51:01.081 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:01.097+0000 7f9ae1c77640 1 -- 172.21.15.88:0/316581927 --> v1:172.21.15.88:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9adc139200 con 0x7f9adc13c040 2024-03-29T10:51:01.082 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:01.097+0000 7f9ae1c77640 10 client.4643.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:51:01.083 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:01.097+0000 7f9ae1c77640 10 monclient: _renew_subs 2024-03-29T10:51:01.083 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:01.098+0000 7f9ae1c77640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-03-29T10:51:01.083 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:01.098+0000 7f9ae1c77640 1 -- 172.21.15.88:0/316581927 --> v1:172.21.15.88:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9adc13e540 con 0x7f9adc13c040 2024-03-29T10:51:01.083 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:01.098+0000 7f9ae1c77640 1 librados: init done 2024-03-29T10:51:01.083 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:01.098+0000 7f9ad97fa640 1 -- 172.21.15.88:0/316581927 <== mon.2 v1:172.21.15.88:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9acc001980 con 0x7f9adc13c040 2024-03-29T10:51:01.083 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:01.098+0000 7f9ad97fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:51:01.083 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:01.098+0000 7f9adaffd640 4 set_mon_vals no callback set 2024-03-29T10:51:01.083 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:01.099+0000 7f9ad97fa640 1 -- 172.21.15.88:0/316581927 <== mon.2 v1:172.21.15.88:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f9acc0037f0 con 0x7f9adc13c040 2024-03-29T10:51:01.083 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:01.099+0000 7f9ad97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:01.083 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:01.099+0000 7f9ad97fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:51:01.083 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:01.099+0000 7f9ad97fa640 10 monclient: dump: 2024-03-29T10:51:01.083 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:51:01.083 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:51:01.084 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:51:01.084 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:51:01.084 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:01.084 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:51:01.084 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:51:01.084 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:51:01.084 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:51:01.084 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:51:01.084 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:01.100+0000 7f9ac2ffd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:51:01.084 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:01.100+0000 7f9ac2ffd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:51:01.084 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:01.100+0000 7f9ac2ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-03-29T10:51:01.084 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:01.100+0000 7f9ac2ffd640 1 -- 172.21.15.88:0/316581927 --> v1:172.21.15.88:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9adc04f730 con 0x7f9adc13c040 2024-03-29T10:51:01.084 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:01.101+0000 7f9ad97fa640 1 -- 172.21.15.88:0/316581927 <== mon.2 v1:172.21.15.88:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4235629220 0 0) 0x7f9acc003950 con 0x7f9adc13c040 2024-03-29T10:51:01.085 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:01.101+0000 7f9ad97fa640 1 -- 172.21.15.88:0/316581927 <== mon.2 v1:172.21.15.88:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1407902845 0 0) 0x7f9acc083800 con 0x7f9adc13c040 2024-03-29T10:51:01.085 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:01.101+0000 7f9ad97fa640 10 client.4643.objecter ms_dispatch 0x7f9adc000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T10:51:01.086 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:01.102+0000 7f9ad97fa640 3 client.4643.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T10:51:01.086 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:01.102+0000 7f9ad97fa640 3 client.4643.objecter handle_osd_map decoding full epoch 11 2024-03-29T10:51:01.086 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:01.102+0000 7f9ad97fa640 20 client.4643.objecter dump_active .. 0 homeless 2024-03-29T10:51:01.086 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:01.102+0000 7f9ae0c75640 10 monclient: get_auth_request con 0x7f9ac806dcc0 auth_method 0 2024-03-29T10:51:01.086 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:01.102+0000 7f9ad97fa640 10 client.4643.objecter ms_handle_connect 0x7f9ac806dcc0 2024-03-29T10:51:01.094 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:01.112+0000 7f9ad97fa640 1 -- 172.21.15.88:0/316581927 <== mon.2 v1:172.21.15.88:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9acc04ef10 con 0x7f9adc13c040 2024-03-29T10:51:01.094 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:01.112+0000 7f9ad97fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:51:01.094 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:01.112+0000 7f9ad97fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:51:01.469 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:01.487+0000 7f9ac2ffd640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:51:01.469 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:01.487+0000 7f9ac2ffd640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:51:01.470 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:01.487+0000 7f9ac2ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-03-29T10:51:01.470 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:01.487+0000 7f9ac2ffd640 1 -- 172.21.15.88:0/316581927 --> v1:172.21.15.88:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9adc0b4700 con 0x7f9adc13c040 2024-03-29T10:51:01.471 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:01.490+0000 7f9ad97fa640 1 -- 172.21.15.88:0/316581927 <== mon.2 v1:172.21.15.88:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 1123663608) 0x7f9acc051010 con 0x7f9adc13c040 2024-03-29T10:51:01.471 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:01.490+0000 7f9ad97fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:51:01.471 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:01.490+0000 7f9ad97fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:51:01.475 INFO:teuthology.misc.health.smithi088.stdout: 2024-03-29T10:51:01.475 INFO:teuthology.misc.health.smithi088.stdout:{"epoch":11,"fsid":"869f5188-f92b-42dd-a46d-8c646e0388a9","created":"2024-03-29T10:40:40.285087+0000","modified":"2024-03-29T10:50:45.236723+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:50:45.236723+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a26c0c1c-8de0-438c-a3f6-3df10ea61574","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"5de24915-8dec-49b6-8b16-f3170197c591","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"ac054af3-d589-4565-9644-e6af75975c53","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"3ec4f045-c714-4a22-bd1a-e07330b45ab3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"cddb9867-8e30-4951-9b13-3d1e78b9c45a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dc6d9911-785d-432f-8b53-56a1ccab302f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"77c88437-b587-496e-8b4d-1824ea8b5305","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ce963d96-5a8e-44c3-99de-c0dbebe234ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:51:01.476 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:01.493+0000 7f9ac2ffd640 10 librados: watch_flush enter 2024-03-29T10:51:01.477 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:01.493+0000 7f9ac2ffd640 10 librados: watch_flush exit 2024-03-29T10:51:01.477 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:01.493+0000 7f9ac2ffd640 20 client.4643.objecter shutdown clearing up homeless session... 2024-03-29T10:51:01.477 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:01.493+0000 7f9ac2ffd640 10 client.4643.objecter successfully canceled tick 2024-03-29T10:51:01.477 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:01.493+0000 7f9ac2ffd640 1 -- 172.21.15.88:0/316581927 >> v1:172.21.15.88:6816/1752861349 conn(0x7f9ac806dcc0 legacy=0x7f9ac8070180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:01.477 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:01.493+0000 7f9ac2ffd640 10 monclient: shutdown 2024-03-29T10:51:01.477 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:01.493+0000 7f9ac2ffd640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:51:01.477 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:01.493+0000 7f9ac2ffd640 1 -- 172.21.15.88:0/316581927 >> v1:172.21.15.88:6790/0 conn(0x7f9adc13c040 legacy=0x7f9adc1e7840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:01.477 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:01.494+0000 7f9ac2ffd640 1 -- 172.21.15.88:0/316581927 shutdown_connections 2024-03-29T10:51:01.477 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:01.494+0000 7f9ac2ffd640 1 -- 172.21.15.88:0/316581927 >> 172.21.15.88:0/316581927 conn(0x7f9adc06d0e0 msgr2=0x7f9adc0b5780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:51:01.477 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:01.494+0000 7f9ac2ffd640 1 -- 172.21.15.88:0/316581927 shutdown_connections 2024-03-29T10:51:01.477 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:01.494+0000 7f9ac2ffd640 1 -- 172.21.15.88:0/316581927 wait complete. 2024-03-29T10:51:01.477 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:01.494+0000 7f9ac2ffd640 1 librados: shutdown 2024-03-29T10:51:01.492 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:51:08.300 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:51:08.455 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:08.473+0000 7f49ea915640 10 monclient: get_monmap_and_config 2024-03-29T10:51:08.455 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:08.473+0000 7f49ea915640 10 monclient: build_initial_monmap 2024-03-29T10:51:08.455 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:08.473+0000 7f49ea915640 10 monclient: monmap: 2024-03-29T10:51:08.455 INFO:teuthology.misc.health.smithi088.stderr:epoch 0 2024-03-29T10:51:08.456 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:51:08.456 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:51:08.474937+0000 2024-03-29T10:51:08.456 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:51:08.474937+0000 2024-03-29T10:51:08.456 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 0 (unknown) 2024-03-29T10:51:08.456 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:51:08.456 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-03-29T10:51:08.456 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-29T10:51:08.456 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-b 2024-03-29T10:51:08.456 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:51:08.457 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:08.475+0000 7f49ea915640 1 Processor -- start 2024-03-29T10:51:08.457 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:08.475+0000 7f49ea915640 1 -- start start 2024-03-29T10:51:08.457 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:08.475+0000 7f49ea915640 10 monclient: init 2024-03-29T10:51:08.458 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:08.476+0000 7f49ea915640 10 monclient: _reopen_session rank -1 2024-03-29T10:51:08.458 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:08.476+0000 7f49ea915640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:51:08.458 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:08.476+0000 7f49ea915640 10 monclient(hunting): picked mon.noname-c con 0x7f49e41457f0 addr v1:172.21.15.196:6789/0 2024-03-29T10:51:08.458 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:08.476+0000 7f49ea915640 10 monclient(hunting): picked mon.noname-b con 0x7f49e413f650 addr v1:172.21.15.88:6790/0 2024-03-29T10:51:08.458 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:08.476+0000 7f49ea915640 10 monclient(hunting): picked mon.noname-a con 0x7f49e40a9d70 addr v1:172.21.15.88:6789/0 2024-03-29T10:51:08.458 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:08.476+0000 7f49ea915640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49e40aaef0 con 0x7f49e40a9d70 2024-03-29T10:51:08.459 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:08.476+0000 7f49ea915640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49e4145bd0 con 0x7f49e41457f0 2024-03-29T10:51:08.459 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:08.476+0000 7f49ea915640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49e4145eb0 con 0x7f49e413f650 2024-03-29T10:51:08.459 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:08.476+0000 7f49ea915640 10 monclient(hunting): _renew_subs 2024-03-29T10:51:08.459 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:08.476+0000 7f49ea915640 10 monclient(hunting): authenticate will time out at 1390.712708s 2024-03-29T10:51:08.459 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:08.476+0000 7f49e3fff640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f49e41457f0 0x7f49e413ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.88:51956/0 (socket says 172.21.15.88:51956) 2024-03-29T10:51:08.459 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:08.477+0000 7f49e3fff640 1 -- 172.21.15.88:0/1470093132 learned_addr learned my addr 172.21.15.88:0/1470093132 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-29T10:51:08.460 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:08.477+0000 7f49e2ffd640 1 -- 172.21.15.88:0/1470093132 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2471812801 0 0) 0x7f49e4145bd0 con 0x7f49e41457f0 2024-03-29T10:51:08.460 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:08.477+0000 7f49e2ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:51:08.460 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:08.477+0000 7f49e2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:08.460 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:08.477+0000 7f49e2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:08.460 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:08.477+0000 7f49e2ffd640 10 monclient(hunting): my global_id is 4573 2024-03-29T10:51:08.460 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:08.477+0000 7f49e2ffd640 1 -- 172.21.15.88:0/1470093132 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f49cc003520 con 0x7f49e41457f0 2024-03-29T10:51:08.460 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:08.477+0000 7f49e2ffd640 1 -- 172.21.15.88:0/1470093132 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3269981284 0 0) 0x7f49e4145eb0 con 0x7f49e413f650 2024-03-29T10:51:08.460 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:08.477+0000 7f49e2ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:51:08.460 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:08.477+0000 7f49e2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:08.460 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:08.477+0000 7f49e2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:08.461 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:08.477+0000 7f49e2ffd640 10 monclient(hunting): my global_id is 4646 2024-03-29T10:51:08.461 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:08.477+0000 7f49e2ffd640 1 -- 172.21.15.88:0/1470093132 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f49e4145bd0 con 0x7f49e413f650 2024-03-29T10:51:08.461 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:08.478+0000 7f49e2ffd640 1 -- 172.21.15.88:0/1470093132 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1896662234 0 0) 0x7f49cc003520 con 0x7f49e41457f0 2024-03-29T10:51:08.461 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:08.478+0000 7f49e2ffd640 1 -- 172.21.15.88:0/1470093132 >> v1:172.21.15.88:6790/0 conn(0x7f49e413f650 legacy=0x7f49e40b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:08.461 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:08.478+0000 7f49e2ffd640 1 -- 172.21.15.88:0/1470093132 >> v1:172.21.15.88:6789/0 conn(0x7f49e40a9d70 legacy=0x7f49e414d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:51:08.461 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:08.478+0000 7f49e2ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:51:08.461 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:08.478+0000 7f49e2ffd640 1 monclient: found mon.noname-c 2024-03-29T10:51:08.461 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:08.478+0000 7f49e2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:51:08.461 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:08.478+0000 7f49e2ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.196:6789/0 2024-03-29T10:51:08.461 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:08.478+0000 7f49e2ffd640 1 -- 172.21.15.88:0/1470093132 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49e4146340 con 0x7f49e41457f0 2024-03-29T10:51:08.461 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:08.478+0000 7f49e2ffd640 10 monclient: _finish_auth 0 2024-03-29T10:51:08.463 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:08.478+0000 7f49e2ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:51:08.464 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:08.478+0000 7f49e2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:51:08.464 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:08.478+0000 7f49ea915640 5 monclient: authenticate success, global_id 4573 2024-03-29T10:51:08.464 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:08.478+0000 7f49ea915640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:51:08.464 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:08.478+0000 7f49e2ffd640 1 -- 172.21.15.88:0/1470093132 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f49d4001690 con 0x7f49e41457f0 2024-03-29T10:51:08.464 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:08.478+0000 7f49e2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:08.464 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:08.478+0000 7f49e2ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:51:08.464 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:08.478+0000 7f49e2ffd640 10 monclient: dump: 2024-03-29T10:51:08.464 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:51:08.464 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:51:08.464 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:51:08.464 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:51:08.464 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:08.464 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:51:08.464 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:51:08.465 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:51:08.465 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:51:08.465 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:51:08.465 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:08.478+0000 7f49ea915640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:51:08.465 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:08.479+0000 7f49e2ffd640 1 -- 172.21.15.88:0/1470093132 <== mon.1 v1:172.21.15.196:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f49d40017f0 con 0x7f49e41457f0 2024-03-29T10:51:08.465 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:08.479+0000 7f49e2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:51:08.465 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:08.479+0000 7f49ea915640 10 monclient: get_monmap_and_config success 2024-03-29T10:51:08.465 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:08.479+0000 7f49ea915640 4 set_mon_vals no callback set 2024-03-29T10:51:08.465 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:08.480+0000 7f49e2ffd640 1 -- 172.21.15.88:0/1470093132 <== mon.1 v1:172.21.15.196:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f49d4001c40 con 0x7f49e41457f0 2024-03-29T10:51:08.465 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:08.480+0000 7f49e2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:08.465 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:08.480+0000 7f49e2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:51:08.465 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:08.480+0000 7f49e2ffd640 10 monclient: dump: 2024-03-29T10:51:08.465 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:51:08.465 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:51:08.466 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:51:08.466 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:51:08.466 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:08.466 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:51:08.466 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:51:08.466 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:51:08.466 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:51:08.466 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:51:08.466 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:08.480+0000 7f49ea915640 10 monclient: shutdown 2024-03-29T10:51:08.466 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:08.480+0000 7f49ea915640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:51:08.466 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:08.480+0000 7f49ea915640 1 -- 172.21.15.88:0/1470093132 >> v1:172.21.15.196:6789/0 conn(0x7f49e41457f0 legacy=0x7f49e413ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:08.466 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:08.481+0000 7f49ea915640 1 -- 172.21.15.88:0/1470093132 shutdown_connections 2024-03-29T10:51:08.466 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:08.481+0000 7f49ea915640 1 -- 172.21.15.88:0/1470093132 >> 172.21.15.88:0/1470093132 conn(0x7f49e4066a00 msgr2=0x7f49e4066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:51:08.466 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:08.481+0000 7f49ea915640 1 -- 172.21.15.88:0/1470093132 shutdown_connections 2024-03-29T10:51:08.467 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:08.481+0000 7f49ea915640 1 -- 172.21.15.88:0/1470093132 wait complete. 2024-03-29T10:51:08.472 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:08.482+0000 7f49ea915640 10 monclient: build_initial_monmap 2024-03-29T10:51:08.472 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:08.482+0000 7f49ea915640 10 monclient: monmap: 2024-03-29T10:51:08.473 INFO:teuthology.misc.health.smithi088.stderr:epoch 0 2024-03-29T10:51:08.473 INFO:teuthology.misc.health.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:51:08.473 INFO:teuthology.misc.health.smithi088.stderr:last_changed 0.000000 2024-03-29T10:51:08.473 INFO:teuthology.misc.health.smithi088.stderr:created 0.000000 2024-03-29T10:51:08.473 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 0 (unknown) 2024-03-29T10:51:08.473 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:51:08.473 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-03-29T10:51:08.473 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-29T10:51:08.473 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-c 2024-03-29T10:51:08.473 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:51:08.473 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:08.483+0000 7f49ea915640 1 librados: starting msgr at 2024-03-29T10:51:08.473 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:08.483+0000 7f49ea915640 1 librados: starting objecter 2024-03-29T10:51:08.473 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:08.483+0000 7f49ea915640 1 Processor -- start 2024-03-29T10:51:08.474 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:08.483+0000 7f49ea915640 1 -- start start 2024-03-29T10:51:08.474 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:08.483+0000 7f49ea915640 1 librados: setting wanted keys 2024-03-29T10:51:08.474 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:08.483+0000 7f49ea915640 1 librados: calling monclient init 2024-03-29T10:51:08.474 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:08.483+0000 7f49ea915640 10 monclient: init 2024-03-29T10:51:08.474 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:08.484+0000 7f49ea915640 10 monclient: _reopen_session rank -1 2024-03-29T10:51:08.474 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:08.484+0000 7f49ea915640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:51:08.474 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:08.484+0000 7f49ea915640 10 monclient(hunting): picked mon.noname-b con 0x7f49e40a9d70 addr v1:172.21.15.196:6789/0 2024-03-29T10:51:08.474 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:08.484+0000 7f49ea915640 10 monclient(hunting): picked mon.noname-a con 0x7f49e413f650 addr v1:172.21.15.88:6789/0 2024-03-29T10:51:08.474 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:08.484+0000 7f49ea915640 10 monclient(hunting): picked mon.noname-c con 0x7f49e41457f0 addr v1:172.21.15.88:6790/0 2024-03-29T10:51:08.474 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:08.484+0000 7f49ea915640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49e406ec10 con 0x7f49e413f650 2024-03-29T10:51:08.474 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:08.484+0000 7f49ea915640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49e406eef0 con 0x7f49e40a9d70 2024-03-29T10:51:08.475 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:08.484+0000 7f49ea915640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49e41db730 con 0x7f49e41457f0 2024-03-29T10:51:08.475 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:08.484+0000 7f49ea915640 10 monclient(hunting): _renew_subs 2024-03-29T10:51:08.475 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:08.484+0000 7f49ea915640 10 monclient(hunting): authenticate will time out at 1390.720509s 2024-03-29T10:51:08.475 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:08.484+0000 7f49e3fff640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f49e40a9d70 0x7f49e413fda0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.88:52722/0 (socket says 172.21.15.88:52722) 2024-03-29T10:51:08.475 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:08.484+0000 7f49e3fff640 1 -- 172.21.15.88:0/1897385351 learned_addr learned my addr 172.21.15.88:0/1897385351 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-29T10:51:08.475 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:08.485+0000 7f49e0ff9640 10 client.?.objecter ms_handle_connect 0x7f49e40a9d70 2024-03-29T10:51:08.475 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:08.485+0000 7f49e0ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:51:08.475 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:08.485+0000 7f49e0ff9640 10 client.?.objecter ms_handle_connect 0x7f49e413f650 2024-03-29T10:51:08.475 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:08.485+0000 7f49e0ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:51:08.475 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:08.485+0000 7f49e0ff9640 10 client.?.objecter ms_handle_connect 0x7f49e41457f0 2024-03-29T10:51:08.475 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:08.485+0000 7f49e0ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:51:08.475 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:08.485+0000 7f49e0ff9640 1 -- 172.21.15.88:0/1897385351 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2142555372 0 0) 0x7f49e406eef0 con 0x7f49e40a9d70 2024-03-29T10:51:08.475 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:08.485+0000 7f49e0ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:51:08.476 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:08.485+0000 7f49e0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:08.476 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:08.485+0000 7f49e0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:08.476 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:08.485+0000 7f49e0ff9640 10 monclient(hunting): my global_id is 4576 2024-03-29T10:51:08.476 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:08.485+0000 7f49e0ff9640 1 -- 172.21.15.88:0/1897385351 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f49d0004eb0 con 0x7f49e40a9d70 2024-03-29T10:51:08.476 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:08.485+0000 7f49e0ff9640 1 -- 172.21.15.88:0/1897385351 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1198872473 0 0) 0x7f49e41db730 con 0x7f49e41457f0 2024-03-29T10:51:08.476 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:08.485+0000 7f49e0ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:51:08.476 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:08.485+0000 7f49e0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:08.476 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:08.485+0000 7f49e0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:08.476 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:08.485+0000 7f49e0ff9640 10 monclient(hunting): my global_id is 4649 2024-03-29T10:51:08.476 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:08.485+0000 7f49e0ff9640 1 -- 172.21.15.88:0/1897385351 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f49e406eef0 con 0x7f49e41457f0 2024-03-29T10:51:08.476 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:08.486+0000 7f49e0ff9640 1 -- 172.21.15.88:0/1897385351 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2494511749 0 0) 0x7f49e406ec10 con 0x7f49e413f650 2024-03-29T10:51:08.476 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:08.486+0000 7f49e0ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:51:08.477 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:08.486+0000 7f49e0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:08.477 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:08.486+0000 7f49e0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:08.477 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:08.486+0000 7f49e0ff9640 10 monclient(hunting): my global_id is 4602 2024-03-29T10:51:08.477 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:08.486+0000 7f49e0ff9640 1 -- 172.21.15.88:0/1897385351 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f49e41db730 con 0x7f49e413f650 2024-03-29T10:51:08.477 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:08.486+0000 7f49e0ff9640 1 -- 172.21.15.88:0/1897385351 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3362147568 0 0) 0x7f49e41db730 con 0x7f49e413f650 2024-03-29T10:51:08.477 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:08.486+0000 7f49e0ff9640 1 -- 172.21.15.88:0/1897385351 --> v1:172.21.15.88:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f49e406ec10 con 0x7f49e413f650 2024-03-29T10:51:08.477 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:08.486+0000 7f49e0ff9640 1 -- 172.21.15.88:0/1897385351 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f49d80016b0 con 0x7f49e413f650 2024-03-29T10:51:08.477 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:08.486+0000 7f49e0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:08.477 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:08.486+0000 7f49e0ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:51:08.477 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:08.486+0000 7f49e0ff9640 10 monclient(hunting): dump: 2024-03-29T10:51:08.477 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:51:08.477 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:51:08.477 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:51:08.477 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:51:08.478 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:08.478 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:51:08.478 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:51:08.478 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:51:08.478 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:51:08.478 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:51:08.478 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:08.486+0000 7f49e0ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:51:08.478 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:08.486+0000 7f49e0ff9640 1 -- 172.21.15.88:0/1897385351 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1293954859 0 0) 0x7f49e406eef0 con 0x7f49e41457f0 2024-03-29T10:51:08.479 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:08.486+0000 7f49e0ff9640 1 -- 172.21.15.88:0/1897385351 --> v1:172.21.15.88:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f49e41db730 con 0x7f49e41457f0 2024-03-29T10:51:08.479 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:08.486+0000 7f49e0ff9640 1 -- 172.21.15.88:0/1897385351 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f49dc001a40 con 0x7f49e41457f0 2024-03-29T10:51:08.479 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:08.486+0000 7f49e0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:08.479 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:08.486+0000 7f49e0ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:51:08.479 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:08.486+0000 7f49e0ff9640 10 monclient(hunting): dump: 2024-03-29T10:51:08.479 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:51:08.479 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:51:08.479 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:51:08.479 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:51:08.479 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:08.479 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:51:08.479 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:51:08.479 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:51:08.479 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:51:08.480 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:51:08.480 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:08.486+0000 7f49e0ff9640 1 -- 172.21.15.88:0/1897385351 <== mon.2 v1:172.21.15.88:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3230233173 0 0) 0x7f49e41db730 con 0x7f49e41457f0 2024-03-29T10:51:08.480 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:08.486+0000 7f49e0ff9640 1 -- 172.21.15.88:0/1897385351 >> v1:172.21.15.196:6789/0 conn(0x7f49e40a9d70 legacy=0x7f49e413fda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:08.480 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:08.486+0000 7f49e0ff9640 1 -- 172.21.15.88:0/1897385351 >> v1:172.21.15.88:6789/0 conn(0x7f49e413f650 legacy=0x7f49e41dc030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:08.480 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:08.486+0000 7f49e0ff9640 10 monclient: _finish_hunting 0 2024-03-29T10:51:08.480 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:08.486+0000 7f49e0ff9640 1 monclient: found mon.c 2024-03-29T10:51:08.480 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:08.486+0000 7f49e0ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:51:08.480 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:08.486+0000 7f49e0ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-03-29T10:51:08.480 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:08.486+0000 7f49e0ff9640 1 -- 172.21.15.88:0/1897385351 --> v1:172.21.15.88:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49e41db8c0 con 0x7f49e41457f0 2024-03-29T10:51:08.480 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:08.486+0000 7f49e0ff9640 10 monclient: _finish_auth 0 2024-03-29T10:51:08.480 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:08.486+0000 7f49e0ff9640 10 monclient: _check_auth_tickets 2024-03-29T10:51:08.480 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:08.486+0000 7f49e0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:51:08.480 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:08.486+0000 7f49ea915640 5 monclient: authenticate success, global_id 4649 2024-03-29T10:51:08.480 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:08.486+0000 7f49ea915640 10 monclient: _renew_subs 2024-03-29T10:51:08.481 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:08.486+0000 7f49ea915640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-03-29T10:51:08.481 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:08.486+0000 7f49ea915640 1 -- 172.21.15.88:0/1897385351 --> v1:172.21.15.88:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49e406de90 con 0x7f49e41457f0 2024-03-29T10:51:08.481 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:08.487+0000 7f49ea915640 10 client.4649.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:51:08.481 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:08.487+0000 7f49ea915640 10 monclient: _renew_subs 2024-03-29T10:51:08.481 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:08.487+0000 7f49ea915640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-03-29T10:51:08.481 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:08.487+0000 7f49ea915640 1 -- 172.21.15.88:0/1897385351 --> v1:172.21.15.88:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f49e413b420 con 0x7f49e41457f0 2024-03-29T10:51:08.481 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:08.487+0000 7f49ea915640 1 librados: init done 2024-03-29T10:51:08.481 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:08.487+0000 7f49e0ff9640 1 -- 172.21.15.88:0/1897385351 <== mon.2 v1:172.21.15.88:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f49dc001f30 con 0x7f49e41457f0 2024-03-29T10:51:08.481 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:08.487+0000 7f49e0ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:51:08.481 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:08.487+0000 7f49e0ff9640 1 -- 172.21.15.88:0/1897385351 <== mon.2 v1:172.21.15.88:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f49dc003b10 con 0x7f49e41457f0 2024-03-29T10:51:08.481 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:08.487+0000 7f49e0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:08.481 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:08.487+0000 7f49e0ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:51:08.481 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:08.487+0000 7f49e0ff9640 10 monclient: dump: 2024-03-29T10:51:08.481 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:51:08.482 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:51:08.482 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:51:08.482 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:51:08.482 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:08.482 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:51:08.482 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:51:08.482 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:51:08.482 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:51:08.482 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:51:08.482 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:08.487+0000 7f49e27fc640 4 set_mon_vals no callback set 2024-03-29T10:51:08.482 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:08.488+0000 7f49ea915640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:51:08.482 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:08.488+0000 7f49ea915640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:51:08.482 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:08.488+0000 7f49ea915640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-03-29T10:51:08.482 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:08.488+0000 7f49ea915640 1 -- 172.21.15.88:0/1897385351 --> v1:172.21.15.88:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49c4005080 con 0x7f49e41457f0 2024-03-29T10:51:08.483 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:08.490+0000 7f49e0ff9640 1 -- 172.21.15.88:0/1897385351 <== mon.2 v1:172.21.15.88:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4235629220 0 0) 0x7f49dc019ae0 con 0x7f49e41457f0 2024-03-29T10:51:08.484 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:08.491+0000 7f49e0ff9640 1 -- 172.21.15.88:0/1897385351 <== mon.2 v1:172.21.15.88:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1407902845 0 0) 0x7f49dc0839e0 con 0x7f49e41457f0 2024-03-29T10:51:08.484 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:08.491+0000 7f49e0ff9640 10 client.4649.objecter ms_dispatch 0x7f49e4000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T10:51:08.484 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:08.491+0000 7f49e0ff9640 3 client.4649.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T10:51:08.484 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:08.491+0000 7f49e0ff9640 3 client.4649.objecter handle_osd_map decoding full epoch 11 2024-03-29T10:51:08.484 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:08.491+0000 7f49e0ff9640 20 client.4649.objecter dump_active .. 0 homeless 2024-03-29T10:51:08.484 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:08.491+0000 7f49e3fff640 10 monclient: get_auth_request con 0x7f49d006e050 auth_method 0 2024-03-29T10:51:08.484 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:08.492+0000 7f49e0ff9640 10 client.4649.objecter ms_handle_connect 0x7f49d006e050 2024-03-29T10:51:08.484 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:08.498+0000 7f49e0ff9640 1 -- 172.21.15.88:0/1897385351 <== mon.2 v1:172.21.15.88:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f49dc04f2c0 con 0x7f49e41457f0 2024-03-29T10:51:08.484 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:08.498+0000 7f49e0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:51:08.485 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:08.498+0000 7f49e0ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:51:08.842 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:08.860+0000 7f49ea915640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:51:08.842 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:08.860+0000 7f49ea915640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:51:08.842 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:08.860+0000 7f49ea915640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-03-29T10:51:08.842 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:08.860+0000 7f49ea915640 1 -- 172.21.15.88:0/1897385351 --> v1:172.21.15.88:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f49c4005330 con 0x7f49e41457f0 2024-03-29T10:51:08.843 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:08.861+0000 7f49e0ff9640 1 -- 172.21.15.88:0/1897385351 <== mon.2 v1:172.21.15.88:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 1123663608) 0x7f49dc0511f0 con 0x7f49e41457f0 2024-03-29T10:51:08.843 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:08.861+0000 7f49e0ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:51:08.843 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:08.861+0000 7f49e0ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:51:08.843 INFO:teuthology.misc.health.smithi088.stdout: 2024-03-29T10:51:08.843 INFO:teuthology.misc.health.smithi088.stdout:{"epoch":11,"fsid":"869f5188-f92b-42dd-a46d-8c646e0388a9","created":"2024-03-29T10:40:40.285087+0000","modified":"2024-03-29T10:50:45.236723+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:50:45.236723+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a26c0c1c-8de0-438c-a3f6-3df10ea61574","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"5de24915-8dec-49b6-8b16-f3170197c591","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"ac054af3-d589-4565-9644-e6af75975c53","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"3ec4f045-c714-4a22-bd1a-e07330b45ab3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"cddb9867-8e30-4951-9b13-3d1e78b9c45a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dc6d9911-785d-432f-8b53-56a1ccab302f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"77c88437-b587-496e-8b4d-1824ea8b5305","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ce963d96-5a8e-44c3-99de-c0dbebe234ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:51:08.846 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:08.865+0000 7f49ea915640 10 librados: watch_flush enter 2024-03-29T10:51:08.847 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:08.865+0000 7f49ea915640 10 librados: watch_flush exit 2024-03-29T10:51:08.847 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:08.865+0000 7f49ea915640 20 client.4649.objecter shutdown clearing up homeless session... 2024-03-29T10:51:08.847 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:08.865+0000 7f49ea915640 10 client.4649.objecter successfully canceled tick 2024-03-29T10:51:08.847 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:08.865+0000 7f49ea915640 1 -- 172.21.15.88:0/1897385351 >> v1:172.21.15.88:6816/1752861349 conn(0x7f49d006e050 legacy=0x7f49d0070510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:08.847 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:08.865+0000 7f49ea915640 10 monclient: shutdown 2024-03-29T10:51:08.847 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:08.865+0000 7f49ea915640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:51:08.847 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:08.865+0000 7f49ea915640 1 -- 172.21.15.88:0/1897385351 >> v1:172.21.15.88:6790/0 conn(0x7f49e41457f0 legacy=0x7f49e41e6170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:08.847 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:08.865+0000 7f49ea915640 1 -- 172.21.15.88:0/1897385351 shutdown_connections 2024-03-29T10:51:08.847 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:08.865+0000 7f49ea915640 1 -- 172.21.15.88:0/1897385351 >> 172.21.15.88:0/1897385351 conn(0x7f49e4066a00 msgr2=0x7f49e40b2520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:51:08.848 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:08.866+0000 7f49ea915640 1 -- 172.21.15.88:0/1897385351 shutdown_connections 2024-03-29T10:51:08.848 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:08.866+0000 7f49ea915640 1 -- 172.21.15.88:0/1897385351 wait complete. 2024-03-29T10:51:08.848 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:08.867+0000 7f49ea915640 1 librados: shutdown 2024-03-29T10:51:08.863 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:51:15.670 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:51:15.824 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:15.842+0000 7f13c0dfd640 10 monclient: get_monmap_and_config 2024-03-29T10:51:15.824 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:15.842+0000 7f13c0dfd640 10 monclient: build_initial_monmap 2024-03-29T10:51:15.824 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:15.842+0000 7f13c0dfd640 10 monclient: monmap: 2024-03-29T10:51:15.824 INFO:teuthology.misc.health.smithi088.stderr:epoch 0 2024-03-29T10:51:15.824 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:51:15.824 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:51:15.843308+0000 2024-03-29T10:51:15.824 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:51:15.843308+0000 2024-03-29T10:51:15.824 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 0 (unknown) 2024-03-29T10:51:15.824 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:51:15.824 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-03-29T10:51:15.825 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-29T10:51:15.825 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-b 2024-03-29T10:51:15.825 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:51:15.825 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:15.843+0000 7f13c0dfd640 1 Processor -- start 2024-03-29T10:51:15.825 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:15.843+0000 7f13c0dfd640 1 -- start start 2024-03-29T10:51:15.825 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:15.843+0000 7f13c0dfd640 10 monclient: init 2024-03-29T10:51:15.826 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:15.844+0000 7f13c0dfd640 10 monclient: _reopen_session rank -1 2024-03-29T10:51:15.826 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:15.844+0000 7f13c0dfd640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:51:15.826 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:15.844+0000 7f13c0dfd640 10 monclient(hunting): picked mon.noname-c con 0x7f13bc0b48a0 addr v1:172.21.15.196:6789/0 2024-03-29T10:51:15.826 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:15.845+0000 7f13c0dfd640 10 monclient(hunting): picked mon.noname-b con 0x7f13bc0b3f70 addr v1:172.21.15.88:6790/0 2024-03-29T10:51:15.827 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:15.845+0000 7f13b3fff640 1 --1- >> v1:172.21.15.88:6790/0 conn(0x7f13bc0b3f70 0x7f13bc13f410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6790/0 says I am v1:172.21.15.88:47842/0 (socket says 172.21.15.88:47842) 2024-03-29T10:51:15.828 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:15.847+0000 7f13c0dfd640 10 monclient(hunting): picked mon.noname-a con 0x7f13bc13fb20 addr v1:172.21.15.88:6789/0 2024-03-29T10:51:15.828 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:15.847+0000 7f13b3fff640 1 -- 172.21.15.88:0/1728336054 learned_addr learned my addr 172.21.15.88:0/1728336054 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-29T10:51:15.828 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:15.847+0000 7f13c0dfd640 1 -- 172.21.15.88:0/1728336054 --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13bc0b43b0 con 0x7f13bc13fb20 2024-03-29T10:51:15.829 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:15.847+0000 7f13c0dfd640 1 -- 172.21.15.88:0/1728336054 --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13bc0b0b90 con 0x7f13bc0b48a0 2024-03-29T10:51:15.829 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:15.847+0000 7f13c0dfd640 1 -- 172.21.15.88:0/1728336054 --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13bc0b19a0 con 0x7f13bc0b3f70 2024-03-29T10:51:15.829 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:15.847+0000 7f13c0dfd640 10 monclient(hunting): _renew_subs 2024-03-29T10:51:15.829 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:15.847+0000 7f13c0dfd640 10 monclient(hunting): authenticate will time out at 1398.083320s 2024-03-29T10:51:15.829 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:15.847+0000 7f13b9d9b640 1 -- 172.21.15.88:0/1728336054 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2794013823 0 0) 0x7f13bc0b19a0 con 0x7f13bc0b3f70 2024-03-29T10:51:15.829 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:15.848+0000 7f13b9d9b640 10 monclient(hunting): _negotiate 2024-03-29T10:51:15.829 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:15.848+0000 7f13b9d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:15.829 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:15.848+0000 7f13b9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:15.829 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:15.848+0000 7f13b9d9b640 10 monclient(hunting): my global_id is 4652 2024-03-29T10:51:15.829 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:15.848+0000 7f13b9d9b640 1 -- 172.21.15.88:0/1728336054 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f13a8003520 con 0x7f13bc0b3f70 2024-03-29T10:51:15.830 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:15.848+0000 7f13b9d9b640 1 -- 172.21.15.88:0/1728336054 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2472867015 0 0) 0x7f13a8003520 con 0x7f13bc0b3f70 2024-03-29T10:51:15.830 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:15.848+0000 7f13b9d9b640 1 -- 172.21.15.88:0/1728336054 >> v1:172.21.15.196:6789/0 conn(0x7f13bc0b48a0 legacy=0x7f13bc0b1290 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:51:15.832 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:15.849+0000 7f13b9d9b640 1 -- 172.21.15.88:0/1728336054 >> v1:172.21.15.88:6789/0 conn(0x7f13bc13fb20 legacy=0x7f13bc14da10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:15.832 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:15.849+0000 7f13b9d9b640 10 monclient: _finish_hunting 0 2024-03-29T10:51:15.832 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:15.849+0000 7f13b9d9b640 1 monclient: found mon.noname-b 2024-03-29T10:51:15.832 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:15.849+0000 7f13b9d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:51:15.832 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:15.849+0000 7f13b9d9b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.88:6790/0 2024-03-29T10:51:15.832 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:15.849+0000 7f13b9d9b640 1 -- 172.21.15.88:0/1728336054 --> v1:172.21.15.88:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13bc0af800 con 0x7f13bc0b3f70 2024-03-29T10:51:15.832 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:15.849+0000 7f13b9d9b640 10 monclient: _finish_auth 0 2024-03-29T10:51:15.832 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:15.849+0000 7f13b9d9b640 10 monclient: _check_auth_tickets 2024-03-29T10:51:15.833 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:15.849+0000 7f13b9d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:51:15.833 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:15.849+0000 7f13b9d9b640 1 -- 172.21.15.88:0/1728336054 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f13a4001690 con 0x7f13bc0b3f70 2024-03-29T10:51:15.833 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:15.849+0000 7f13b9d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:15.833 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:15.849+0000 7f13b9d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:51:15.833 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:15.849+0000 7f13b9d9b640 10 monclient: dump: 2024-03-29T10:51:15.833 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:51:15.833 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:51:15.833 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:51:15.833 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:51:15.833 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:15.833 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:51:15.833 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:51:15.833 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:51:15.833 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:51:15.833 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:51:15.834 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:15.849+0000 7f13c0dfd640 5 monclient: authenticate success, global_id 4652 2024-03-29T10:51:15.834 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:15.849+0000 7f13c0dfd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:51:15.834 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:15.852+0000 7f13b9d9b640 1 -- 172.21.15.88:0/1728336054 <== mon.2 v1:172.21.15.88:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f13a4001860 con 0x7f13bc0b3f70 2024-03-29T10:51:15.834 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:15.852+0000 7f13b9d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:51:15.834 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:15.852+0000 7f13c0dfd640 10 monclient: get_monmap_and_config success 2024-03-29T10:51:15.835 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:15.852+0000 7f13c0dfd640 4 set_mon_vals no callback set 2024-03-29T10:51:15.835 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:15.853+0000 7f13c0dfd640 10 monclient: shutdown 2024-03-29T10:51:15.835 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:15.853+0000 7f13c0dfd640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:51:15.835 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:15.853+0000 7f13c0dfd640 1 -- 172.21.15.88:0/1728336054 >> v1:172.21.15.88:6790/0 conn(0x7f13bc0b3f70 legacy=0x7f13bc13f410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:15.836 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:15.854+0000 7f13c0dfd640 1 -- 172.21.15.88:0/1728336054 shutdown_connections 2024-03-29T10:51:15.836 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:15.854+0000 7f13c0dfd640 1 -- 172.21.15.88:0/1728336054 >> 172.21.15.88:0/1728336054 conn(0x7f13bc06c8a0 msgr2=0x7f13bc06cc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:51:15.837 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:15.855+0000 7f13c0dfd640 1 -- 172.21.15.88:0/1728336054 shutdown_connections 2024-03-29T10:51:15.837 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:15.856+0000 7f13c0dfd640 1 -- 172.21.15.88:0/1728336054 wait complete. 2024-03-29T10:51:15.839 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:15.857+0000 7f13c0dfd640 10 monclient: build_initial_monmap 2024-03-29T10:51:15.839 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:15.857+0000 7f13c0dfd640 10 monclient: monmap: 2024-03-29T10:51:15.839 INFO:teuthology.misc.health.smithi088.stderr:epoch 0 2024-03-29T10:51:15.839 INFO:teuthology.misc.health.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:51:15.839 INFO:teuthology.misc.health.smithi088.stderr:last_changed 0.000000 2024-03-29T10:51:15.839 INFO:teuthology.misc.health.smithi088.stderr:created 0.000000 2024-03-29T10:51:15.839 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 0 (unknown) 2024-03-29T10:51:15.839 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:51:15.839 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-03-29T10:51:15.839 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-29T10:51:15.839 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-c 2024-03-29T10:51:15.839 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:51:15.840 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:15.858+0000 7f13c0dfd640 1 librados: starting msgr at 2024-03-29T10:51:15.840 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:15.858+0000 7f13c0dfd640 1 librados: starting objecter 2024-03-29T10:51:15.840 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:15.858+0000 7f13c0dfd640 1 Processor -- start 2024-03-29T10:51:15.841 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:15.859+0000 7f13c0dfd640 1 -- start start 2024-03-29T10:51:15.841 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:15.859+0000 7f13c0dfd640 1 librados: setting wanted keys 2024-03-29T10:51:15.841 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:15.859+0000 7f13c0dfd640 1 librados: calling monclient init 2024-03-29T10:51:15.841 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:15.859+0000 7f13c0dfd640 10 monclient: init 2024-03-29T10:51:15.842 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:15.860+0000 7f13c0dfd640 10 monclient: _reopen_session rank -1 2024-03-29T10:51:15.842 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:15.860+0000 7f13c0dfd640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:51:15.842 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:15.860+0000 7f13c0dfd640 10 monclient(hunting): picked mon.noname-a con 0x7f13bc0b3f70 addr v1:172.21.15.88:6789/0 2024-03-29T10:51:15.842 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:15.860+0000 7f13c0dfd640 10 monclient(hunting): picked mon.noname-b con 0x7f13bc0b48a0 addr v1:172.21.15.196:6789/0 2024-03-29T10:51:15.842 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:15.860+0000 7f13c0dfd640 10 monclient(hunting): picked mon.noname-c con 0x7f13bc13fb20 addr v1:172.21.15.88:6790/0 2024-03-29T10:51:15.842 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:15.860+0000 7f13c0dfd640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13bc1393b0 con 0x7f13bc0b3f70 2024-03-29T10:51:15.842 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:15.860+0000 7f13c0dfd640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13bc139690 con 0x7f13bc0b48a0 2024-03-29T10:51:15.842 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:15.860+0000 7f13c0dfd640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13bc138190 con 0x7f13bc13fb20 2024-03-29T10:51:15.842 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:15.860+0000 7f13c0dfd640 10 monclient(hunting): _renew_subs 2024-03-29T10:51:15.843 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:15.860+0000 7f13c0dfd640 10 monclient(hunting): authenticate will time out at 1398.096615s 2024-03-29T10:51:15.845 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:15.864+0000 7f13bad9d640 1 --1- >> v1:172.21.15.88:6790/0 conn(0x7f13bc13fb20 0x7f13bc1e19c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6790/0 says I am v1:172.21.15.88:51670/0 (socket says 172.21.15.88:51670) 2024-03-29T10:51:15.846 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:15.864+0000 7f13bad9d640 1 -- 172.21.15.88:0/784879156 learned_addr learned my addr 172.21.15.88:0/784879156 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-29T10:51:15.846 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:15.864+0000 7f13b3fff640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f13bc0b48a0 0x7f13bc139e00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.88:45457/0 (socket says 172.21.15.88:45457) 2024-03-29T10:51:15.846 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:15.864+0000 7f13b2ffd640 10 client.?.objecter ms_handle_connect 0x7f13bc13fb20 2024-03-29T10:51:15.846 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:15.864+0000 7f13b2ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:51:15.846 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:15.864+0000 7f13b2ffd640 10 client.?.objecter ms_handle_connect 0x7f13bc0b48a0 2024-03-29T10:51:15.846 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:15.864+0000 7f13b2ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:51:15.847 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:15.864+0000 7f13b2ffd640 1 -- 172.21.15.88:0/784879156 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 769855822 0 0) 0x7f13bc138190 con 0x7f13bc13fb20 2024-03-29T10:51:15.847 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:15.864+0000 7f13b2ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:51:15.847 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:15.864+0000 7f13b2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:15.847 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:15.864+0000 7f13b2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:15.847 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:15.864+0000 7f13b2ffd640 10 monclient(hunting): my global_id is 4655 2024-03-29T10:51:15.847 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:15.864+0000 7f13b2ffd640 1 -- 172.21.15.88:0/784879156 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f139c004eb0 con 0x7f13bc13fb20 2024-03-29T10:51:15.847 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:15.865+0000 7f13b2ffd640 10 client.?.objecter ms_handle_connect 0x7f13bc0b3f70 2024-03-29T10:51:15.847 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:15.865+0000 7f13b2ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:51:15.847 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:15.865+0000 7f13b2ffd640 1 -- 172.21.15.88:0/784879156 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2993848156 0 0) 0x7f13bc139690 con 0x7f13bc0b48a0 2024-03-29T10:51:15.848 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:15.865+0000 7f13b2ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:51:15.848 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:15.865+0000 7f13b2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:15.848 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:15.865+0000 7f13b2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:15.848 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:15.865+0000 7f13b2ffd640 10 monclient(hunting): my global_id is 4579 2024-03-29T10:51:15.848 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:15.865+0000 7f13b2ffd640 1 -- 172.21.15.88:0/784879156 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f13bc138190 con 0x7f13bc0b48a0 2024-03-29T10:51:15.848 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:15.865+0000 7f13b2ffd640 1 -- 172.21.15.88:0/784879156 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3040829156 0 0) 0x7f13bc1393b0 con 0x7f13bc0b3f70 2024-03-29T10:51:15.848 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:15.865+0000 7f13b2ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:51:15.848 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:15.865+0000 7f13b2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:15.848 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:15.865+0000 7f13b2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:15.848 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:15.865+0000 7f13b2ffd640 10 monclient(hunting): my global_id is 4608 2024-03-29T10:51:15.848 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:15.865+0000 7f13b2ffd640 1 -- 172.21.15.88:0/784879156 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f13bc139690 con 0x7f13bc0b3f70 2024-03-29T10:51:15.848 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:15.865+0000 7f13b2ffd640 1 -- 172.21.15.88:0/784879156 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 740091497 0 0) 0x7f139c004eb0 con 0x7f13bc13fb20 2024-03-29T10:51:15.848 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:15.865+0000 7f13b2ffd640 1 -- 172.21.15.88:0/784879156 --> v1:172.21.15.88:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f13bc1393b0 con 0x7f13bc13fb20 2024-03-29T10:51:15.848 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:15.865+0000 7f13b2ffd640 1 -- 172.21.15.88:0/784879156 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f13b4001c70 con 0x7f13bc13fb20 2024-03-29T10:51:15.849 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:15.865+0000 7f13b2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:15.849 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:15.865+0000 7f13b2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:51:15.849 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:15.865+0000 7f13b2ffd640 10 monclient(hunting): dump: 2024-03-29T10:51:15.849 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:51:15.849 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:51:15.849 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:51:15.849 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:51:15.849 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:15.849 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:51:15.849 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:51:15.849 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:51:15.849 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:51:15.849 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:51:15.849 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:15.865+0000 7f13b2ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:51:15.849 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:15.865+0000 7f13b2ffd640 1 -- 172.21.15.88:0/784879156 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1652016650 0 0) 0x7f13bc139690 con 0x7f13bc0b3f70 2024-03-29T10:51:15.850 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:15.865+0000 7f13b2ffd640 1 -- 172.21.15.88:0/784879156 --> v1:172.21.15.88:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f139c004eb0 con 0x7f13bc0b3f70 2024-03-29T10:51:15.850 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:15.865+0000 7f13b2ffd640 1 -- 172.21.15.88:0/784879156 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f13ac0017d0 con 0x7f13bc0b3f70 2024-03-29T10:51:15.850 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:15.865+0000 7f13b2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:15.850 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:15.865+0000 7f13b2ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:51:15.850 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:15.865+0000 7f13b2ffd640 10 monclient(hunting): dump: 2024-03-29T10:51:15.850 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:51:15.850 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:51:15.850 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:51:15.850 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:51:15.850 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:15.850 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:51:15.850 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:51:15.850 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:51:15.850 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:51:15.851 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:51:15.851 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:15.867+0000 7f13b2ffd640 1 -- 172.21.15.88:0/784879156 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 57459395 0 0) 0x7f13bc138190 con 0x7f13bc0b48a0 2024-03-29T10:51:15.852 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:15.867+0000 7f13b2ffd640 1 -- 172.21.15.88:0/784879156 --> v1:172.21.15.196:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f13bc139690 con 0x7f13bc0b48a0 2024-03-29T10:51:15.852 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:15.867+0000 7f13b2ffd640 1 -- 172.21.15.88:0/784879156 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f13a4004940 con 0x7f13bc0b48a0 2024-03-29T10:51:15.852 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:15.867+0000 7f13b2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:15.852 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:15.868+0000 7f13b2ffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T10:51:15.852 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:15.868+0000 7f13b2ffd640 10 monclient(hunting): dump: 2024-03-29T10:51:15.852 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:51:15.852 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:51:15.852 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:51:15.852 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:51:15.852 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:15.852 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:51:15.852 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:51:15.852 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:51:15.852 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:51:15.853 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:51:15.853 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:15.868+0000 7f13b2ffd640 1 -- 172.21.15.88:0/784879156 <== mon.0 v1:172.21.15.88:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3691131627 0 0) 0x7f139c004eb0 con 0x7f13bc0b3f70 2024-03-29T10:51:15.853 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:15.868+0000 7f13b2ffd640 1 -- 172.21.15.88:0/784879156 >> v1:172.21.15.88:6790/0 conn(0x7f13bc13fb20 legacy=0x7f13bc1e19c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:15.853 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:15.868+0000 7f13b2ffd640 1 -- 172.21.15.88:0/784879156 >> v1:172.21.15.196:6789/0 conn(0x7f13bc0b48a0 legacy=0x7f13bc139e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:15.853 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:15.868+0000 7f13b2ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:51:15.853 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:15.868+0000 7f13b2ffd640 1 monclient: found mon.a 2024-03-29T10:51:15.853 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:15.868+0000 7f13b2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:51:15.853 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:15.868+0000 7f13b2ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-03-29T10:51:15.853 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:15.868+0000 7f13b2ffd640 1 -- 172.21.15.88:0/784879156 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13bc1386b0 con 0x7f13bc0b3f70 2024-03-29T10:51:15.853 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:15.868+0000 7f13b2ffd640 10 monclient: _finish_auth 0 2024-03-29T10:51:15.853 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:15.868+0000 7f13b2ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:51:15.853 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:15.868+0000 7f13b2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:51:15.853 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:15.868+0000 7f13c0dfd640 5 monclient: authenticate success, global_id 4608 2024-03-29T10:51:15.853 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:15.868+0000 7f13c0dfd640 10 monclient: _renew_subs 2024-03-29T10:51:15.854 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:15.868+0000 7f13c0dfd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-03-29T10:51:15.854 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:15.868+0000 7f13c0dfd640 1 -- 172.21.15.88:0/784879156 --> v1:172.21.15.88:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13bc137d40 con 0x7f13bc0b3f70 2024-03-29T10:51:15.854 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:15.868+0000 7f13c0dfd640 10 client.4608.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:51:15.854 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:15.868+0000 7f13c0dfd640 10 monclient: _renew_subs 2024-03-29T10:51:15.854 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:15.868+0000 7f13c0dfd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-03-29T10:51:15.854 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:15.868+0000 7f13c0dfd640 1 -- 172.21.15.88:0/784879156 --> v1:172.21.15.88:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f13bc13a6f0 con 0x7f13bc0b3f70 2024-03-29T10:51:15.854 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:15.868+0000 7f13c0dfd640 1 librados: init done 2024-03-29T10:51:15.854 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:15.870+0000 7f13c0dfd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:51:15.854 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:15.870+0000 7f13c0dfd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:51:15.854 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:15.870+0000 7f13c0dfd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-03-29T10:51:15.854 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:15.870+0000 7f13c0dfd640 1 -- 172.21.15.88:0/784879156 --> v1:172.21.15.88:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13bc138320 con 0x7f13bc0b3f70 2024-03-29T10:51:15.855 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:15.870+0000 7f13b2ffd640 1 -- 172.21.15.88:0/784879156 <== mon.0 v1:172.21.15.88:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f13ac001930 con 0x7f13bc0b3f70 2024-03-29T10:51:15.855 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:15.870+0000 7f13b2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:51:15.855 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:15.870+0000 7f13b2ffd640 1 -- 172.21.15.88:0/784879156 <== mon.0 v1:172.21.15.88:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f13ac003790 con 0x7f13bc0b3f70 2024-03-29T10:51:15.855 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:15.870+0000 7f13b2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:15.855 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:15.870+0000 7f13b2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:51:15.855 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:15.870+0000 7f13b2ffd640 10 monclient: dump: 2024-03-29T10:51:15.855 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:51:15.855 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:51:15.855 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:51:15.855 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:51:15.855 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:15.855 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:51:15.856 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:51:15.856 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:51:15.856 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:51:15.856 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:51:15.856 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:15.871+0000 7f13b959a640 4 set_mon_vals no callback set 2024-03-29T10:51:15.857 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:15.872+0000 7f13b2ffd640 1 -- 172.21.15.88:0/784879156 <== mon.0 v1:172.21.15.88:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4235629220 0 0) 0x7f13ac019760 con 0x7f13bc0b3f70 2024-03-29T10:51:15.857 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:15.873+0000 7f13b2ffd640 1 -- 172.21.15.88:0/784879156 <== mon.0 v1:172.21.15.88:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1407902845 0 0) 0x7f13ac083660 con 0x7f13bc0b3f70 2024-03-29T10:51:15.857 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:15.875+0000 7f13b3fff640 10 monclient: get_auth_request con 0x7f139c06e300 auth_method 0 2024-03-29T10:51:15.857 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:15.876+0000 7f13b2ffd640 10 client.4608.objecter ms_dispatch 0x7f13bc000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T10:51:15.858 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:15.876+0000 7f13b2ffd640 3 client.4608.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T10:51:15.858 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:15.876+0000 7f13b2ffd640 3 client.4608.objecter handle_osd_map decoding full epoch 11 2024-03-29T10:51:15.858 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:15.876+0000 7f13b2ffd640 20 client.4608.objecter dump_active .. 0 homeless 2024-03-29T10:51:15.858 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:15.876+0000 7f13b2ffd640 10 client.4608.objecter ms_handle_connect 0x7f139c06e300 2024-03-29T10:51:15.861 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:15.880+0000 7f13b2ffd640 1 -- 172.21.15.88:0/784879156 <== mon.0 v1:172.21.15.88:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f13ac04ef40 con 0x7f13bc0b3f70 2024-03-29T10:51:15.861 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:15.880+0000 7f13b2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:51:15.862 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:15.880+0000 7f13b2ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:51:16.182 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:16.200+0000 7f13c0dfd640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:51:16.182 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:16.200+0000 7f13c0dfd640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:51:16.182 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:16.200+0000 7f13c0dfd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-03-29T10:51:16.182 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:16.200+0000 7f13c0dfd640 1 -- 172.21.15.88:0/784879156 --> v1:172.21.15.88:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f13bc075f10 con 0x7f13bc0b3f70 2024-03-29T10:51:16.183 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:16.201+0000 7f13b2ffd640 1 -- 172.21.15.88:0/784879156 <== mon.0 v1:172.21.15.88:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 1123663608) 0x7f13ac050e70 con 0x7f13bc0b3f70 2024-03-29T10:51:16.183 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:16.201+0000 7f13b2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:51:16.183 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:16.201+0000 7f13b2ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:51:16.183 INFO:teuthology.misc.health.smithi088.stdout: 2024-03-29T10:51:16.183 INFO:teuthology.misc.health.smithi088.stdout:{"epoch":11,"fsid":"869f5188-f92b-42dd-a46d-8c646e0388a9","created":"2024-03-29T10:40:40.285087+0000","modified":"2024-03-29T10:50:45.236723+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:50:45.236723+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a26c0c1c-8de0-438c-a3f6-3df10ea61574","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"5de24915-8dec-49b6-8b16-f3170197c591","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"ac054af3-d589-4565-9644-e6af75975c53","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"3ec4f045-c714-4a22-bd1a-e07330b45ab3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"cddb9867-8e30-4951-9b13-3d1e78b9c45a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dc6d9911-785d-432f-8b53-56a1ccab302f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"77c88437-b587-496e-8b4d-1824ea8b5305","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ce963d96-5a8e-44c3-99de-c0dbebe234ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:51:16.186 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:16.205+0000 7f13c0dfd640 10 librados: watch_flush enter 2024-03-29T10:51:16.187 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:16.205+0000 7f13c0dfd640 10 librados: watch_flush exit 2024-03-29T10:51:16.187 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:16.205+0000 7f13c0dfd640 20 client.4608.objecter shutdown clearing up homeless session... 2024-03-29T10:51:16.187 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:16.205+0000 7f13c0dfd640 10 client.4608.objecter successfully canceled tick 2024-03-29T10:51:16.187 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:16.205+0000 7f13c0dfd640 1 -- 172.21.15.88:0/784879156 >> v1:172.21.15.88:6816/1752861349 conn(0x7f139c06e300 legacy=0x7f139c0707c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:16.187 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:16.205+0000 7f13c0dfd640 10 monclient: shutdown 2024-03-29T10:51:16.187 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:16.205+0000 7f13c0dfd640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:51:16.187 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:16.205+0000 7f13c0dfd640 1 -- 172.21.15.88:0/784879156 >> v1:172.21.15.88:6789/0 conn(0x7f13bc0b3f70 legacy=0x7f13bc137580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:16.187 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:16.205+0000 7f13c0dfd640 1 -- 172.21.15.88:0/784879156 shutdown_connections 2024-03-29T10:51:16.187 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:16.205+0000 7f13c0dfd640 1 -- 172.21.15.88:0/784879156 >> 172.21.15.88:0/784879156 conn(0x7f13bc06c8a0 msgr2=0x7f13bc06cc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:51:16.188 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:16.206+0000 7f13c0dfd640 1 -- 172.21.15.88:0/784879156 shutdown_connections 2024-03-29T10:51:16.188 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:16.206+0000 7f13c0dfd640 1 -- 172.21.15.88:0/784879156 wait complete. 2024-03-29T10:51:16.188 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:16.206+0000 7f13c0dfd640 1 librados: shutdown 2024-03-29T10:51:16.204 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:51:23.011 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:51:23.143 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:23.162+0000 7f0521b6b640 10 monclient: get_monmap_and_config 2024-03-29T10:51:23.144 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:23.162+0000 7f0521b6b640 10 monclient: build_initial_monmap 2024-03-29T10:51:23.144 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:23.162+0000 7f0521b6b640 10 monclient: monmap: 2024-03-29T10:51:23.144 INFO:teuthology.misc.health.smithi088.stderr:epoch 0 2024-03-29T10:51:23.144 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:51:23.144 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:51:23.163324+0000 2024-03-29T10:51:23.144 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:51:23.163324+0000 2024-03-29T10:51:23.144 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 0 (unknown) 2024-03-29T10:51:23.144 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:51:23.144 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-03-29T10:51:23.144 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-29T10:51:23.144 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-b 2024-03-29T10:51:23.144 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:51:23.145 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:23.163+0000 7f0521b6b640 1 Processor -- start 2024-03-29T10:51:23.145 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:23.163+0000 7f0521b6b640 1 -- start start 2024-03-29T10:51:23.145 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:23.163+0000 7f0521b6b640 10 monclient: init 2024-03-29T10:51:23.147 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:23.164+0000 7f0521b6b640 10 monclient: _reopen_session rank -1 2024-03-29T10:51:23.147 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:23.164+0000 7f0521b6b640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:51:23.147 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:23.164+0000 7f0521b6b640 10 monclient(hunting): picked mon.noname-c con 0x7f051c1422b0 addr v1:172.21.15.196:6789/0 2024-03-29T10:51:23.147 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:23.164+0000 7f0521b6b640 10 monclient(hunting): picked mon.noname-b con 0x7f051c142690 addr v1:172.21.15.88:6790/0 2024-03-29T10:51:23.147 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:23.164+0000 7f0521b6b640 10 monclient(hunting): picked mon.noname-a con 0x7f051c0b17d0 addr v1:172.21.15.88:6789/0 2024-03-29T10:51:23.147 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:23.164+0000 7f0521b6b640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f051c142c80 con 0x7f051c0b17d0 2024-03-29T10:51:23.147 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:23.164+0000 7f0521b6b640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f051c0aae40 con 0x7f051c1422b0 2024-03-29T10:51:23.147 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:23.164+0000 7f0521b6b640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f051c13fbd0 con 0x7f051c142690 2024-03-29T10:51:23.147 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:23.164+0000 7f051affd640 1 --1- >> v1:172.21.15.88:6790/0 conn(0x7f051c142690 0x7f051c0b2590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6790/0 says I am v1:172.21.15.88:58050/0 (socket says 172.21.15.88:58050) 2024-03-29T10:51:23.147 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:23.164+0000 7f051affd640 1 -- 172.21.15.88:0/3733760144 learned_addr learned my addr 172.21.15.88:0/3733760144 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-29T10:51:23.147 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:23.164+0000 7f0521b6b640 10 monclient(hunting): _renew_subs 2024-03-29T10:51:23.147 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:23.164+0000 7f0521b6b640 10 monclient(hunting): authenticate will time out at 1405.400545s 2024-03-29T10:51:23.151 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:23.165+0000 7f051a7fc640 1 -- 172.21.15.88:0/3733760144 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2718803901 0 0) 0x7f051c13fbd0 con 0x7f051c142690 2024-03-29T10:51:23.151 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:23.166+0000 7f051a7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:51:23.151 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:23.166+0000 7f051a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:23.152 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:23.166+0000 7f051a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:23.152 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:23.166+0000 7f051a7fc640 10 monclient(hunting): my global_id is 4658 2024-03-29T10:51:23.152 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:23.166+0000 7f051a7fc640 1 -- 172.21.15.88:0/3733760144 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0504003520 con 0x7f051c142690 2024-03-29T10:51:23.152 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:23.166+0000 7f051a7fc640 1 -- 172.21.15.88:0/3733760144 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2367965511 0 0) 0x7f051c0aae40 con 0x7f051c1422b0 2024-03-29T10:51:23.152 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:23.166+0000 7f051a7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:51:23.152 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:23.166+0000 7f051a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:23.152 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:23.166+0000 7f051a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:23.152 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:23.166+0000 7f051a7fc640 10 monclient(hunting): my global_id is 4582 2024-03-29T10:51:23.152 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:23.166+0000 7f051a7fc640 1 -- 172.21.15.88:0/3733760144 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f051c13fbd0 con 0x7f051c1422b0 2024-03-29T10:51:23.153 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:23.166+0000 7f051a7fc640 1 -- 172.21.15.88:0/3733760144 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1789358803 0 0) 0x7f0504003520 con 0x7f051c142690 2024-03-29T10:51:23.153 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:23.166+0000 7f051a7fc640 1 -- 172.21.15.88:0/3733760144 >> v1:172.21.15.196:6789/0 conn(0x7f051c1422b0 legacy=0x7f051c13f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:23.153 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:23.166+0000 7f051a7fc640 1 -- 172.21.15.88:0/3733760144 >> v1:172.21.15.88:6789/0 conn(0x7f051c0b17d0 legacy=0x7f051c0b2ca0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:51:23.153 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:23.168+0000 7f051a7fc640 10 monclient: _finish_hunting 0 2024-03-29T10:51:23.153 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:23.168+0000 7f051a7fc640 1 monclient: found mon.noname-b 2024-03-29T10:51:23.153 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:23.168+0000 7f051a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:51:23.153 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:23.168+0000 7f051a7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.88:6790/0 2024-03-29T10:51:23.153 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:23.168+0000 7f051a7fc640 1 -- 172.21.15.88:0/3733760144 --> v1:172.21.15.88:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f051c151940 con 0x7f051c142690 2024-03-29T10:51:23.153 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:23.168+0000 7f051a7fc640 10 monclient: _finish_auth 0 2024-03-29T10:51:23.153 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:23.168+0000 7f051a7fc640 10 monclient: _check_auth_tickets 2024-03-29T10:51:23.153 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:23.168+0000 7f051a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:51:23.153 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:23.168+0000 7f051a7fc640 1 -- 172.21.15.88:0/3733760144 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f0510001690 con 0x7f051c142690 2024-03-29T10:51:23.153 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:23.168+0000 7f051a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:23.154 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:23.168+0000 7f051a7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:51:23.154 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:23.168+0000 7f051a7fc640 10 monclient: dump: 2024-03-29T10:51:23.154 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:51:23.154 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:51:23.154 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:51:23.154 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:51:23.154 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:23.154 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:51:23.154 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:51:23.154 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:51:23.154 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:51:23.154 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:51:23.154 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:23.168+0000 7f0521b6b640 5 monclient: authenticate success, global_id 4658 2024-03-29T10:51:23.154 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:23.168+0000 7f0521b6b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:51:23.155 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:23.168+0000 7f051a7fc640 1 -- 172.21.15.88:0/3733760144 <== mon.2 v1:172.21.15.88:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f05100017f0 con 0x7f051c142690 2024-03-29T10:51:23.155 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:23.168+0000 7f051a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:51:23.155 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:23.168+0000 7f051a7fc640 1 -- 172.21.15.88:0/3733760144 <== mon.2 v1:172.21.15.88:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f0510001bc0 con 0x7f051c142690 2024-03-29T10:51:23.155 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:23.168+0000 7f0521b6b640 10 monclient: get_monmap_and_config success 2024-03-29T10:51:23.155 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:23.168+0000 7f0521b6b640 4 set_mon_vals no callback set 2024-03-29T10:51:23.155 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:23.169+0000 7f051a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:23.155 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:23.169+0000 7f051a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:51:23.155 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:23.169+0000 7f051a7fc640 10 monclient: dump: 2024-03-29T10:51:23.155 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:51:23.155 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:51:23.155 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:51:23.155 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:51:23.155 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:23.155 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:51:23.156 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:51:23.156 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:51:23.156 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:51:23.156 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:51:23.156 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:23.169+0000 7f0521b6b640 10 monclient: shutdown 2024-03-29T10:51:23.156 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:23.169+0000 7f0521b6b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:51:23.156 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:23.169+0000 7f0521b6b640 1 -- 172.21.15.88:0/3733760144 >> v1:172.21.15.88:6790/0 conn(0x7f051c142690 legacy=0x7f051c0b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:23.156 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:23.169+0000 7f0521b6b640 1 -- 172.21.15.88:0/3733760144 shutdown_connections 2024-03-29T10:51:23.156 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:23.169+0000 7f0521b6b640 1 -- 172.21.15.88:0/3733760144 >> 172.21.15.88:0/3733760144 conn(0x7f051c066a00 msgr2=0x7f051c066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:51:23.156 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:23.170+0000 7f0521b6b640 1 -- 172.21.15.88:0/3733760144 shutdown_connections 2024-03-29T10:51:23.156 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:23.170+0000 7f0521b6b640 1 -- 172.21.15.88:0/3733760144 wait complete. 2024-03-29T10:51:23.162 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:23.171+0000 7f0521b6b640 10 monclient: build_initial_monmap 2024-03-29T10:51:23.162 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:23.171+0000 7f0521b6b640 10 monclient: monmap: 2024-03-29T10:51:23.162 INFO:teuthology.misc.health.smithi088.stderr:epoch 0 2024-03-29T10:51:23.162 INFO:teuthology.misc.health.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:51:23.162 INFO:teuthology.misc.health.smithi088.stderr:last_changed 0.000000 2024-03-29T10:51:23.162 INFO:teuthology.misc.health.smithi088.stderr:created 0.000000 2024-03-29T10:51:23.162 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 0 (unknown) 2024-03-29T10:51:23.163 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:51:23.163 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-03-29T10:51:23.163 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-29T10:51:23.163 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-c 2024-03-29T10:51:23.163 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:51:23.163 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:23.171+0000 7f0521b6b640 1 librados: starting msgr at 2024-03-29T10:51:23.163 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:23.171+0000 7f0521b6b640 1 librados: starting objecter 2024-03-29T10:51:23.163 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:23.171+0000 7f0521b6b640 1 Processor -- start 2024-03-29T10:51:23.163 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:23.172+0000 7f0521b6b640 1 -- start start 2024-03-29T10:51:23.163 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:23.172+0000 7f0521b6b640 1 librados: setting wanted keys 2024-03-29T10:51:23.163 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:23.172+0000 7f0521b6b640 1 librados: calling monclient init 2024-03-29T10:51:23.163 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:23.172+0000 7f0521b6b640 10 monclient: init 2024-03-29T10:51:23.163 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:23.173+0000 7f0521b6b640 10 monclient: _reopen_session rank -1 2024-03-29T10:51:23.164 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:23.173+0000 7f0521b6b640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:51:23.164 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:23.174+0000 7f0521b6b640 10 monclient(hunting): picked mon.noname-b con 0x7f051c0b17d0 addr v1:172.21.15.196:6789/0 2024-03-29T10:51:23.164 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:23.174+0000 7f0521b6b640 10 monclient(hunting): picked mon.noname-c con 0x7f051c1422b0 addr v1:172.21.15.88:6790/0 2024-03-29T10:51:23.164 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:23.174+0000 7f0521b6b640 10 monclient(hunting): picked mon.noname-a con 0x7f051c0b7610 addr v1:172.21.15.88:6789/0 2024-03-29T10:51:23.164 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:23.174+0000 7f0521b6b640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f051c0ac1f0 con 0x7f051c0b7610 2024-03-29T10:51:23.164 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:23.174+0000 7f0521b6b640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f051c0b5a10 con 0x7f051c0b17d0 2024-03-29T10:51:23.164 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:23.174+0000 7f051affd640 1 --1- >> v1:172.21.15.88:6790/0 conn(0x7f051c1422b0 0x7f051c0b0980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6790/0 says I am v1:172.21.15.88:59094/0 (socket says 172.21.15.88:59094) 2024-03-29T10:51:23.164 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:23.174+0000 7f0521b6b640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f051c0b5ba0 con 0x7f051c1422b0 2024-03-29T10:51:23.164 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:23.174+0000 7f0521b6b640 10 monclient(hunting): _renew_subs 2024-03-29T10:51:23.164 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:23.174+0000 7f051affd640 1 -- 172.21.15.88:0/1253426871 learned_addr learned my addr 172.21.15.88:0/1253426871 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-29T10:51:23.164 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:23.174+0000 7f0521b6b640 10 monclient(hunting): authenticate will time out at 1405.410235s 2024-03-29T10:51:23.164 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:23.174+0000 7f0503fff640 10 client.?.objecter ms_handle_connect 0x7f051c1422b0 2024-03-29T10:51:23.164 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:23.174+0000 7f0503fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:51:23.165 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:23.174+0000 7f0503fff640 10 client.?.objecter ms_handle_connect 0x7f051c0b17d0 2024-03-29T10:51:23.165 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:23.174+0000 7f0503fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:51:23.165 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:23.174+0000 7f0503fff640 1 -- 172.21.15.88:0/1253426871 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 630410641 0 0) 0x7f051c0b5ba0 con 0x7f051c1422b0 2024-03-29T10:51:23.165 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:23.174+0000 7f0503fff640 10 monclient(hunting): _negotiate 2024-03-29T10:51:23.165 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:23.174+0000 7f0503fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:23.165 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:23.174+0000 7f0503fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:23.165 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:23.175+0000 7f0503fff640 10 monclient(hunting): my global_id is 4661 2024-03-29T10:51:23.165 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:23.175+0000 7f0503fff640 1 -- 172.21.15.88:0/1253426871 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0508004eb0 con 0x7f051c1422b0 2024-03-29T10:51:23.165 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:23.175+0000 7f0503fff640 1 -- 172.21.15.88:0/1253426871 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 803277907 0 0) 0x7f051c0b5a10 con 0x7f051c0b17d0 2024-03-29T10:51:23.165 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:23.175+0000 7f0503fff640 10 monclient(hunting): _negotiate 2024-03-29T10:51:23.165 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:23.175+0000 7f0503fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:23.165 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:23.175+0000 7f0503fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:23.165 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:23.175+0000 7f0503fff640 10 monclient(hunting): my global_id is 4585 2024-03-29T10:51:23.166 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:23.175+0000 7f0503fff640 1 -- 172.21.15.88:0/1253426871 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f051c0b5ba0 con 0x7f051c0b17d0 2024-03-29T10:51:23.166 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:23.175+0000 7f0503fff640 1 -- 172.21.15.88:0/1253426871 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3418223935 0 0) 0x7f0508004eb0 con 0x7f051c1422b0 2024-03-29T10:51:23.166 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:23.175+0000 7f0503fff640 1 -- 172.21.15.88:0/1253426871 --> v1:172.21.15.88:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f051c0b5a10 con 0x7f051c1422b0 2024-03-29T10:51:23.166 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:23.175+0000 7f0503fff640 1 -- 172.21.15.88:0/1253426871 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f0510001b00 con 0x7f051c1422b0 2024-03-29T10:51:23.166 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:23.175+0000 7f0503fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:23.166 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:23.175+0000 7f0503fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:51:23.166 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:23.175+0000 7f0503fff640 10 monclient(hunting): dump: 2024-03-29T10:51:23.166 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:51:23.166 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:51:23.166 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:51:23.166 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:51:23.166 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:23.166 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:51:23.166 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:51:23.167 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:51:23.167 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:51:23.167 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:51:23.167 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:23.175+0000 7f0503fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:51:23.167 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:23.175+0000 7f0503fff640 1 -- 172.21.15.88:0/1253426871 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4162384572 0 0) 0x7f051c0b5ba0 con 0x7f051c0b17d0 2024-03-29T10:51:23.167 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:23.175+0000 7f0503fff640 1 -- 172.21.15.88:0/1253426871 --> v1:172.21.15.196:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0508004eb0 con 0x7f051c0b17d0 2024-03-29T10:51:23.167 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:23.175+0000 7f0503fff640 1 -- 172.21.15.88:0/1253426871 <== mon.2 v1:172.21.15.88:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2856401792 0 0) 0x7f051c0b5a10 con 0x7f051c1422b0 2024-03-29T10:51:23.167 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:23.175+0000 7f0503fff640 1 -- 172.21.15.88:0/1253426871 >> v1:172.21.15.196:6789/0 conn(0x7f051c0b17d0 legacy=0x7f051c0b5300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:23.167 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:23.175+0000 7f0503fff640 1 -- 172.21.15.88:0/1253426871 >> v1:172.21.15.88:6789/0 conn(0x7f051c0b7610 legacy=0x7f051c1e2d40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:51:23.167 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:23.176+0000 7f0503fff640 10 monclient: _finish_hunting 0 2024-03-29T10:51:23.167 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:23.176+0000 7f0503fff640 1 monclient: found mon.c 2024-03-29T10:51:23.167 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:23.176+0000 7f0503fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:51:23.167 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:23.176+0000 7f0503fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-03-29T10:51:23.167 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:23.176+0000 7f0503fff640 1 -- 172.21.15.88:0/1253426871 --> v1:172.21.15.88:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f051c132280 con 0x7f051c1422b0 2024-03-29T10:51:23.168 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:23.176+0000 7f0503fff640 10 monclient: _finish_auth 0 2024-03-29T10:51:23.168 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:23.176+0000 7f0503fff640 10 monclient: _check_auth_tickets 2024-03-29T10:51:23.168 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:23.176+0000 7f0503fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:51:23.168 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:23.176+0000 7f0521b6b640 5 monclient: authenticate success, global_id 4661 2024-03-29T10:51:23.168 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:23.176+0000 7f0521b6b640 10 monclient: _renew_subs 2024-03-29T10:51:23.168 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:23.176+0000 7f0521b6b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-03-29T10:51:23.168 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:23.176+0000 7f0521b6b640 1 -- 172.21.15.88:0/1253426871 --> v1:172.21.15.88:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f051c0b8050 con 0x7f051c1422b0 2024-03-29T10:51:23.168 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:23.176+0000 7f0521b6b640 10 client.4661.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:51:23.168 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:23.176+0000 7f0521b6b640 10 monclient: _renew_subs 2024-03-29T10:51:23.168 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:23.176+0000 7f0521b6b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-03-29T10:51:23.168 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:23.176+0000 7f0521b6b640 1 -- 172.21.15.88:0/1253426871 --> v1:172.21.15.88:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f051c0b7180 con 0x7f051c1422b0 2024-03-29T10:51:23.168 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:23.176+0000 7f0521b6b640 1 librados: init done 2024-03-29T10:51:23.168 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:23.176+0000 7f0503fff640 1 -- 172.21.15.88:0/1253426871 <== mon.2 v1:172.21.15.88:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0510004a90 con 0x7f051c1422b0 2024-03-29T10:51:23.168 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:23.176+0000 7f0503fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:51:23.169 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:23.176+0000 7f0519ffb640 4 set_mon_vals no callback set 2024-03-29T10:51:23.169 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:23.177+0000 7f0503fff640 1 -- 172.21.15.88:0/1253426871 <== mon.2 v1:172.21.15.88:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f0510004da0 con 0x7f051c1422b0 2024-03-29T10:51:23.169 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:23.177+0000 7f0503fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:23.169 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:23.177+0000 7f0503fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:51:23.169 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:23.177+0000 7f0503fff640 10 monclient: dump: 2024-03-29T10:51:23.169 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:51:23.169 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:51:23.169 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:51:23.169 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:51:23.169 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:23.169 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:51:23.169 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:51:23.169 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:51:23.169 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:51:23.169 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:51:23.169 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:23.178+0000 7f0521b6b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:51:23.170 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:23.178+0000 7f0521b6b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:51:23.170 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:23.178+0000 7f0521b6b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-03-29T10:51:23.170 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:23.178+0000 7f0521b6b640 1 -- 172.21.15.88:0/1253426871 --> v1:172.21.15.88:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04fc005080 con 0x7f051c1422b0 2024-03-29T10:51:23.170 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:23.179+0000 7f0503fff640 1 -- 172.21.15.88:0/1253426871 <== mon.2 v1:172.21.15.88:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4235629220 0 0) 0x7f051001ad70 con 0x7f051c1422b0 2024-03-29T10:51:23.170 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:23.179+0000 7f0503fff640 1 -- 172.21.15.88:0/1253426871 <== mon.2 v1:172.21.15.88:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1407902845 0 0) 0x7f0510084a50 con 0x7f051c1422b0 2024-03-29T10:51:23.170 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:23.179+0000 7f0503fff640 10 client.4661.objecter ms_dispatch 0x7f051c000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T10:51:23.170 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:23.179+0000 7f0503fff640 3 client.4661.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T10:51:23.170 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:23.179+0000 7f0503fff640 3 client.4661.objecter handle_osd_map decoding full epoch 11 2024-03-29T10:51:23.170 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:23.179+0000 7f0503fff640 20 client.4661.objecter dump_active .. 0 homeless 2024-03-29T10:51:23.170 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:23.179+0000 7f051b7fe640 10 monclient: get_auth_request con 0x7f050806df40 auth_method 0 2024-03-29T10:51:23.170 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:23.180+0000 7f0503fff640 10 client.4661.objecter ms_handle_connect 0x7f050806df40 2024-03-29T10:51:23.171 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:23.187+0000 7f0503fff640 1 -- 172.21.15.88:0/1253426871 <== mon.2 v1:172.21.15.88:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0510050330 con 0x7f051c1422b0 2024-03-29T10:51:23.171 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:23.187+0000 7f0503fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:51:23.171 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:23.187+0000 7f0503fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:51:23.478 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:23.497+0000 7f0521b6b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:51:23.478 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:23.497+0000 7f0521b6b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:51:23.479 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:23.497+0000 7f0521b6b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-03-29T10:51:23.479 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:23.497+0000 7f0521b6b640 1 -- 172.21.15.88:0/1253426871 --> v1:172.21.15.88:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f04fc005330 con 0x7f051c1422b0 2024-03-29T10:51:23.480 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:23.498+0000 7f0503fff640 1 -- 172.21.15.88:0/1253426871 <== mon.2 v1:172.21.15.88:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 1123663608) 0x7f0510052260 con 0x7f051c1422b0 2024-03-29T10:51:23.480 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:23.499+0000 7f0503fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:51:23.480 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:23.499+0000 7f0503fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:51:23.481 INFO:teuthology.misc.health.smithi088.stdout: 2024-03-29T10:51:23.481 INFO:teuthology.misc.health.smithi088.stdout:{"epoch":11,"fsid":"869f5188-f92b-42dd-a46d-8c646e0388a9","created":"2024-03-29T10:40:40.285087+0000","modified":"2024-03-29T10:50:45.236723+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:50:45.236723+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a26c0c1c-8de0-438c-a3f6-3df10ea61574","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"5de24915-8dec-49b6-8b16-f3170197c591","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"ac054af3-d589-4565-9644-e6af75975c53","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"3ec4f045-c714-4a22-bd1a-e07330b45ab3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"cddb9867-8e30-4951-9b13-3d1e78b9c45a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dc6d9911-785d-432f-8b53-56a1ccab302f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"77c88437-b587-496e-8b4d-1824ea8b5305","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ce963d96-5a8e-44c3-99de-c0dbebe234ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:51:23.484 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:23.502+0000 7f0521b6b640 10 librados: watch_flush enter 2024-03-29T10:51:23.484 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:23.502+0000 7f0521b6b640 10 librados: watch_flush exit 2024-03-29T10:51:23.484 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:23.502+0000 7f0521b6b640 20 client.4661.objecter shutdown clearing up homeless session... 2024-03-29T10:51:23.485 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:23.502+0000 7f0521b6b640 10 client.4661.objecter successfully canceled tick 2024-03-29T10:51:23.485 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:23.502+0000 7f0521b6b640 1 -- 172.21.15.88:0/1253426871 >> v1:172.21.15.88:6816/1752861349 conn(0x7f050806df40 legacy=0x7f0508070400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:23.485 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:23.502+0000 7f0521b6b640 10 monclient: shutdown 2024-03-29T10:51:23.485 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:23.502+0000 7f0521b6b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:51:23.485 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:23.502+0000 7f0521b6b640 1 -- 172.21.15.88:0/1253426871 >> v1:172.21.15.88:6790/0 conn(0x7f051c1422b0 legacy=0x7f051c0b0980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:23.485 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:23.502+0000 7f0521b6b640 1 -- 172.21.15.88:0/1253426871 shutdown_connections 2024-03-29T10:51:23.485 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:23.502+0000 7f0521b6b640 1 -- 172.21.15.88:0/1253426871 >> 172.21.15.88:0/1253426871 conn(0x7f051c066a00 msgr2=0x7f051c13e150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:51:23.485 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:23.503+0000 7f0521b6b640 1 -- 172.21.15.88:0/1253426871 shutdown_connections 2024-03-29T10:51:23.485 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:23.503+0000 7f0521b6b640 1 -- 172.21.15.88:0/1253426871 wait complete. 2024-03-29T10:51:23.485 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:23.503+0000 7f0521b6b640 1 librados: shutdown 2024-03-29T10:51:23.500 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:51:30.307 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:51:30.451 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:30.469+0000 7f565edcf640 10 monclient: get_monmap_and_config 2024-03-29T10:51:30.451 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:30.469+0000 7f565edcf640 10 monclient: build_initial_monmap 2024-03-29T10:51:30.451 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:30.469+0000 7f565edcf640 10 monclient: monmap: 2024-03-29T10:51:30.452 INFO:teuthology.misc.health.smithi088.stderr:epoch 0 2024-03-29T10:51:30.452 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:51:30.452 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:51:30.470646+0000 2024-03-29T10:51:30.452 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:51:30.470646+0000 2024-03-29T10:51:30.452 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 0 (unknown) 2024-03-29T10:51:30.452 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:51:30.452 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-03-29T10:51:30.452 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-29T10:51:30.452 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-b 2024-03-29T10:51:30.452 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:51:30.453 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:30.470+0000 7f565edcf640 1 Processor -- start 2024-03-29T10:51:30.453 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:30.470+0000 7f565edcf640 1 -- start start 2024-03-29T10:51:30.453 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:30.470+0000 7f565edcf640 10 monclient: init 2024-03-29T10:51:30.453 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:30.471+0000 7f565edcf640 10 monclient: _reopen_session rank -1 2024-03-29T10:51:30.454 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:30.471+0000 7f565edcf640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:51:30.454 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:30.471+0000 7f565edcf640 10 monclient(hunting): picked mon.noname-a con 0x7f56580b4920 addr v1:172.21.15.88:6789/0 2024-03-29T10:51:30.454 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:30.471+0000 7f565edcf640 10 monclient(hunting): picked mon.noname-b con 0x7f56580b1790 addr v1:172.21.15.88:6790/0 2024-03-29T10:51:30.454 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:30.471+0000 7f565edcf640 10 monclient(hunting): picked mon.noname-c con 0x7f5658146790 addr v1:172.21.15.196:6789/0 2024-03-29T10:51:30.454 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:30.472+0000 7f5657fff640 1 --1- >> v1:172.21.15.88:6790/0 conn(0x7f56580b1790 0x7f56580b1bb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6790/0 says I am v1:172.21.15.88:58104/0 (socket says 172.21.15.88:58104) 2024-03-29T10:51:30.454 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:30.472+0000 7f5657fff640 1 -- 172.21.15.88:0/1086276334 learned_addr learned my addr 172.21.15.88:0/1086276334 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-29T10:51:30.454 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:30.472+0000 7f565edcf640 1 -- 172.21.15.88:0/1086276334 --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56580b4070 con 0x7f56580b4920 2024-03-29T10:51:30.454 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:30.472+0000 7f565edcf640 1 -- 172.21.15.88:0/1086276334 --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f565806fb80 con 0x7f5658146790 2024-03-29T10:51:30.454 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:30.472+0000 7f565edcf640 1 -- 172.21.15.88:0/1086276334 --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56580b42c0 con 0x7f56580b1790 2024-03-29T10:51:30.454 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:30.472+0000 7f565edcf640 10 monclient(hunting): _renew_subs 2024-03-29T10:51:30.454 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:30.472+0000 7f565edcf640 10 monclient(hunting): authenticate will time out at 1412.708364s 2024-03-29T10:51:30.458 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:30.473+0000 7f5657fff640 1 -- 172.21.15.88:0/1086276334 >> v1:172.21.15.88:6790/0 conn(0x7f56580b1790 legacy=0x7f56580b1bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2024-03-29T10:51:30.458 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:30.473+0000 7f5657fff640 1 -- 172.21.15.88:0/1086276334 >> v1:172.21.15.88:6790/0 conn(0x7f56580b1790 legacy=0x7f56580b1bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-29T10:51:30.458 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:30.473+0000 7f5657fff640 1 --1- 172.21.15.88:0/1086276334 >> v1:172.21.15.88:6790/0 conn(0x7f56580b1790 0x7f56580b1bb0 :-1 s=OPENED pgs=1050047 cs=1 l=1).handle_message read tag failed 2024-03-29T10:51:30.458 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:30.473+0000 7f5657fff640 1 --1- 172.21.15.88:0/1086276334 >> v1:172.21.15.88:6790/0 conn(0x7f56580b1790 0x7f56580b1bb0 :-1 s=OPENED pgs=1050047 cs=1 l=1).fault on lossy channel, failing 2024-03-29T10:51:30.458 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:30.473+0000 7f56577fe640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.88:6790/0 2024-03-29T10:51:30.458 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:30.473+0000 7f56577fe640 1 -- 172.21.15.88:0/1086276334 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3083427708 0 0) 0x7f56580b4070 con 0x7f56580b4920 2024-03-29T10:51:30.459 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:30.473+0000 7f56577fe640 10 monclient(hunting): _negotiate 2024-03-29T10:51:30.459 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:30.473+0000 7f56577fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:30.459 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:30.473+0000 7f56577fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:30.459 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:30.473+0000 7f56577fe640 10 monclient(hunting): my global_id is 4611 2024-03-29T10:51:30.459 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:30.473+0000 7f56577fe640 1 -- 172.21.15.88:0/1086276334 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5640003520 con 0x7f56580b4920 2024-03-29T10:51:30.459 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:30.473+0000 7f56577fe640 1 -- 172.21.15.88:0/1086276334 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 527491362 0 0) 0x7f565806fb80 con 0x7f5658146790 2024-03-29T10:51:30.459 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:30.473+0000 7f56577fe640 10 monclient(hunting): _negotiate 2024-03-29T10:51:30.459 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:30.473+0000 7f56577fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:30.459 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:30.473+0000 7f56577fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:30.459 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:30.473+0000 7f56577fe640 10 monclient(hunting): my global_id is 4588 2024-03-29T10:51:30.459 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:30.473+0000 7f56577fe640 1 -- 172.21.15.88:0/1086276334 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f56580b4070 con 0x7f5658146790 2024-03-29T10:51:30.459 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:30.473+0000 7f56577fe640 1 -- 172.21.15.88:0/1086276334 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3262363154 0 0) 0x7f5640003520 con 0x7f56580b4920 2024-03-29T10:51:30.459 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:30.473+0000 7f56577fe640 1 -- 172.21.15.88:0/1086276334 >> v1:172.21.15.88:6790/0 conn(0x7f56580b1790 legacy=0x7f56580b1bb0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-29T10:51:30.460 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:30.473+0000 7f56577fe640 1 -- 172.21.15.88:0/1086276334 >> v1:172.21.15.196:6789/0 conn(0x7f5658146790 legacy=0x7f5658146c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:30.460 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:30.473+0000 7f56577fe640 10 monclient: _finish_hunting 0 2024-03-29T10:51:30.460 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:30.473+0000 7f56577fe640 1 monclient: found mon.noname-a 2024-03-29T10:51:30.460 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:30.473+0000 7f56577fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:51:30.460 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:30.473+0000 7f56577fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.88:6789/0 2024-03-29T10:51:30.460 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:30.473+0000 7f56577fe640 1 -- 172.21.15.88:0/1086276334 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56580b22c0 con 0x7f56580b4920 2024-03-29T10:51:30.460 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:30.473+0000 7f56577fe640 10 monclient: _finish_auth 0 2024-03-29T10:51:30.460 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:30.473+0000 7f56577fe640 10 monclient: _check_auth_tickets 2024-03-29T10:51:30.460 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:30.473+0000 7f56577fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:51:30.460 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:30.473+0000 7f56577fe640 1 -- 172.21.15.88:0/1086276334 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f564c001a10 con 0x7f56580b4920 2024-03-29T10:51:30.460 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:30.473+0000 7f56577fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:30.460 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:30.473+0000 7f56577fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:51:30.460 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:30.473+0000 7f56577fe640 10 monclient: dump: 2024-03-29T10:51:30.461 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:51:30.461 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:51:30.461 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:51:30.461 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:51:30.461 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:30.461 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:51:30.461 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:51:30.461 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:51:30.461 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:51:30.461 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:51:30.461 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:30.473+0000 7f565edcf640 5 monclient: authenticate success, global_id 4611 2024-03-29T10:51:30.461 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:30.473+0000 7f565edcf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:51:30.461 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:30.474+0000 7f56577fe640 1 -- 172.21.15.88:0/1086276334 <== mon.0 v1:172.21.15.88:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f564c001b70 con 0x7f56580b4920 2024-03-29T10:51:30.462 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:30.474+0000 7f56577fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:51:30.462 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:30.474+0000 7f56577fe640 1 -- 172.21.15.88:0/1086276334 <== mon.0 v1:172.21.15.88:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f564c001f80 con 0x7f56580b4920 2024-03-29T10:51:30.462 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:30.474+0000 7f56577fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:30.462 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:30.474+0000 7f56577fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:51:30.462 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:30.474+0000 7f56577fe640 10 monclient: dump: 2024-03-29T10:51:30.462 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:51:30.462 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:51:30.462 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:51:30.462 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:51:30.462 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:30.462 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:51:30.462 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:51:30.462 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:51:30.462 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:51:30.463 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:51:30.463 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:30.474+0000 7f565edcf640 10 monclient: get_monmap_and_config success 2024-03-29T10:51:30.463 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:30.474+0000 7f565edcf640 4 set_mon_vals no callback set 2024-03-29T10:51:30.463 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:30.475+0000 7f565edcf640 10 monclient: shutdown 2024-03-29T10:51:30.463 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:30.475+0000 7f565edcf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:51:30.463 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:30.475+0000 7f565edcf640 1 -- 172.21.15.88:0/1086276334 >> v1:172.21.15.88:6789/0 conn(0x7f56580b4920 legacy=0x7f565806e510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:30.463 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:30.475+0000 7f565edcf640 1 -- 172.21.15.88:0/1086276334 shutdown_connections 2024-03-29T10:51:30.463 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:30.475+0000 7f565edcf640 1 -- 172.21.15.88:0/1086276334 >> 172.21.15.88:0/1086276334 conn(0x7f5658066a00 msgr2=0x7f5658066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:51:30.463 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:30.475+0000 7f565edcf640 1 -- 172.21.15.88:0/1086276334 shutdown_connections 2024-03-29T10:51:30.463 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:30.475+0000 7f565edcf640 1 -- 172.21.15.88:0/1086276334 wait complete. 2024-03-29T10:51:30.463 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:30.476+0000 7f565edcf640 10 monclient: build_initial_monmap 2024-03-29T10:51:30.463 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:30.476+0000 7f565edcf640 10 monclient: monmap: 2024-03-29T10:51:30.463 INFO:teuthology.misc.health.smithi088.stderr:epoch 0 2024-03-29T10:51:30.463 INFO:teuthology.misc.health.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:51:30.464 INFO:teuthology.misc.health.smithi088.stderr:last_changed 0.000000 2024-03-29T10:51:30.464 INFO:teuthology.misc.health.smithi088.stderr:created 0.000000 2024-03-29T10:51:30.464 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 0 (unknown) 2024-03-29T10:51:30.464 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:51:30.464 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-03-29T10:51:30.464 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-29T10:51:30.464 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-c 2024-03-29T10:51:30.464 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:51:30.464 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:30.477+0000 7f565edcf640 1 librados: starting msgr at 2024-03-29T10:51:30.464 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:30.477+0000 7f565edcf640 1 librados: starting objecter 2024-03-29T10:51:30.464 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:30.477+0000 7f565edcf640 1 Processor -- start 2024-03-29T10:51:30.471 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:30.477+0000 7f565edcf640 1 -- start start 2024-03-29T10:51:30.471 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:30.477+0000 7f565edcf640 1 librados: setting wanted keys 2024-03-29T10:51:30.471 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:30.477+0000 7f565edcf640 1 librados: calling monclient init 2024-03-29T10:51:30.472 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:30.477+0000 7f565edcf640 10 monclient: init 2024-03-29T10:51:30.472 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:30.478+0000 7f565edcf640 10 monclient: _reopen_session rank -1 2024-03-29T10:51:30.472 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:30.478+0000 7f565edcf640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:51:30.472 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:30.478+0000 7f565edcf640 10 monclient(hunting): picked mon.noname-b con 0x7f56580b1790 addr v1:172.21.15.196:6789/0 2024-03-29T10:51:30.472 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:30.478+0000 7f565edcf640 10 monclient(hunting): picked mon.noname-c con 0x7f56580b4920 addr v1:172.21.15.88:6790/0 2024-03-29T10:51:30.472 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:30.478+0000 7f565edcf640 10 monclient(hunting): picked mon.noname-a con 0x7f5658146790 addr v1:172.21.15.88:6789/0 2024-03-29T10:51:30.472 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:30.478+0000 7f565edcf640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56580aad20 con 0x7f5658146790 2024-03-29T10:51:30.472 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:30.478+0000 7f565edcf640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56580abd90 con 0x7f56580b1790 2024-03-29T10:51:30.472 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:30.478+0000 7f565edcf640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56580ac070 con 0x7f56580b4920 2024-03-29T10:51:30.472 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:30.478+0000 7f565edcf640 10 monclient(hunting): _renew_subs 2024-03-29T10:51:30.472 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:30.478+0000 7f565edcf640 10 monclient(hunting): authenticate will time out at 1412.714586s 2024-03-29T10:51:30.472 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:30.478+0000 7f5657fff640 1 --1- >> v1:172.21.15.88:6790/0 conn(0x7f56580b4920 0x7f56581e30d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6790/0 says I am v1:172.21.15.88:59840/0 (socket says 172.21.15.88:59840) 2024-03-29T10:51:30.472 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:30.478+0000 7f5657fff640 1 -- 172.21.15.88:0/71405952 learned_addr learned my addr 172.21.15.88:0/71405952 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-29T10:51:30.473 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:30.479+0000 7f56557fa640 10 client.?.objecter ms_handle_connect 0x7f56580b4920 2024-03-29T10:51:30.473 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:30.479+0000 7f56557fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:51:30.473 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:30.479+0000 7f56557fa640 10 client.?.objecter ms_handle_connect 0x7f5658146790 2024-03-29T10:51:30.473 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:30.479+0000 7f56557fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:51:30.473 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:30.479+0000 7f56557fa640 1 -- 172.21.15.88:0/71405952 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4170400336 0 0) 0x7f56580ac070 con 0x7f56580b4920 2024-03-29T10:51:30.473 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:30.479+0000 7f56557fa640 10 monclient(hunting): _negotiate 2024-03-29T10:51:30.473 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:30.479+0000 7f56557fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:30.473 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:30.479+0000 7f56557fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:30.473 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:30.479+0000 7f56557fa640 10 monclient(hunting): my global_id is 4664 2024-03-29T10:51:30.473 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:30.479+0000 7f56557fa640 1 -- 172.21.15.88:0/71405952 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5644004eb0 con 0x7f56580b4920 2024-03-29T10:51:30.473 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:30.479+0000 7f56557fa640 10 client.?.objecter ms_handle_connect 0x7f56580b1790 2024-03-29T10:51:30.473 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:30.479+0000 7f56557fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:51:30.473 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:30.479+0000 7f56557fa640 1 -- 172.21.15.88:0/71405952 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1365549613 0 0) 0x7f56580aad20 con 0x7f5658146790 2024-03-29T10:51:30.473 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:30.479+0000 7f56557fa640 10 monclient(hunting): _negotiate 2024-03-29T10:51:30.473 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:30.479+0000 7f56557fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:30.474 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:30.479+0000 7f56557fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:30.474 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:30.479+0000 7f56557fa640 10 monclient(hunting): my global_id is 4614 2024-03-29T10:51:30.474 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:30.479+0000 7f56557fa640 1 -- 172.21.15.88:0/71405952 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f56580ac070 con 0x7f5658146790 2024-03-29T10:51:30.474 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:30.479+0000 7f56557fa640 1 -- 172.21.15.88:0/71405952 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1842538434 0 0) 0x7f5644004eb0 con 0x7f56580b4920 2024-03-29T10:51:30.474 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:30.479+0000 7f56557fa640 1 -- 172.21.15.88:0/71405952 --> v1:172.21.15.88:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f56580aad20 con 0x7f56580b4920 2024-03-29T10:51:30.474 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:30.480+0000 7f56557fa640 1 -- 172.21.15.88:0/71405952 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f56480019f0 con 0x7f56580b4920 2024-03-29T10:51:30.474 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:30.480+0000 7f56557fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:30.474 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:30.480+0000 7f56557fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:51:30.474 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:30.480+0000 7f56557fa640 10 monclient(hunting): dump: 2024-03-29T10:51:30.474 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:51:30.474 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:51:30.474 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:51:30.474 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:51:30.474 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:30.475 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:51:30.475 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:51:30.475 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:51:30.475 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:51:30.475 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:51:30.475 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:30.480+0000 7f56557fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:51:30.475 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:30.480+0000 7f56557fa640 1 -- 172.21.15.88:0/71405952 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1782450077 0 0) 0x7f56580abd90 con 0x7f56580b1790 2024-03-29T10:51:30.475 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:30.480+0000 7f56557fa640 10 monclient(hunting): _negotiate 2024-03-29T10:51:30.475 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:30.480+0000 7f56557fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:30.475 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:30.480+0000 7f56557fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:30.475 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:30.480+0000 7f56557fa640 10 monclient(hunting): my global_id is 4591 2024-03-29T10:51:30.475 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:30.480+0000 7f56557fa640 1 -- 172.21.15.88:0/71405952 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5644004eb0 con 0x7f56580b1790 2024-03-29T10:51:30.475 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:30.480+0000 7f56557fa640 1 -- 172.21.15.88:0/71405952 <== mon.2 v1:172.21.15.88:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3590536462 0 0) 0x7f56580aad20 con 0x7f56580b4920 2024-03-29T10:51:30.475 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:30.480+0000 7f56557fa640 1 -- 172.21.15.88:0/71405952 >> v1:172.21.15.196:6789/0 conn(0x7f56580b1790 legacy=0x7f5658142600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:30.476 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:30.480+0000 7f56557fa640 1 -- 172.21.15.88:0/71405952 >> v1:172.21.15.88:6789/0 conn(0x7f5658146790 legacy=0x7f56581e6800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:30.476 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:30.480+0000 7f56557fa640 10 monclient: _finish_hunting 0 2024-03-29T10:51:30.476 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:30.480+0000 7f56557fa640 1 monclient: found mon.c 2024-03-29T10:51:30.476 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:30.480+0000 7f56557fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:51:30.476 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:30.480+0000 7f56557fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-03-29T10:51:30.476 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:30.480+0000 7f56557fa640 1 -- 172.21.15.88:0/71405952 --> v1:172.21.15.88:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f565813d1c0 con 0x7f56580b4920 2024-03-29T10:51:30.476 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:30.480+0000 7f56557fa640 10 monclient: _finish_auth 0 2024-03-29T10:51:30.476 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:30.480+0000 7f56557fa640 10 monclient: _check_auth_tickets 2024-03-29T10:51:30.476 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:30.480+0000 7f56557fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:51:30.476 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:30.480+0000 7f565edcf640 5 monclient: authenticate success, global_id 4664 2024-03-29T10:51:30.476 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:30.480+0000 7f565edcf640 10 monclient: _renew_subs 2024-03-29T10:51:30.476 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:30.480+0000 7f565edcf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-03-29T10:51:30.476 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:30.480+0000 7f565edcf640 1 -- 172.21.15.88:0/71405952 --> v1:172.21.15.88:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56580a9a10 con 0x7f56580b4920 2024-03-29T10:51:30.476 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:30.480+0000 7f565edcf640 10 client.4664.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:51:30.477 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:30.480+0000 7f565edcf640 10 monclient: _renew_subs 2024-03-29T10:51:30.477 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:30.480+0000 7f565edcf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-03-29T10:51:30.477 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:30.480+0000 7f565edcf640 1 -- 172.21.15.88:0/71405952 --> v1:172.21.15.88:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f56580aa800 con 0x7f56580b4920 2024-03-29T10:51:30.477 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:30.480+0000 7f565edcf640 1 librados: init done 2024-03-29T10:51:30.477 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:30.481+0000 7f56557fa640 1 -- 172.21.15.88:0/71405952 <== mon.2 v1:172.21.15.88:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f56480047e0 con 0x7f56580b4920 2024-03-29T10:51:30.477 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:30.481+0000 7f56557fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:51:30.477 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:30.481+0000 7f56557fa640 1 -- 172.21.15.88:0/71405952 <== mon.2 v1:172.21.15.88:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f5648004bf0 con 0x7f56580b4920 2024-03-29T10:51:30.477 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:30.481+0000 7f56557fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:30.477 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:30.481+0000 7f56557fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:51:30.477 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:30.481+0000 7f56557fa640 10 monclient: dump: 2024-03-29T10:51:30.478 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:51:30.478 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:51:30.478 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:51:30.478 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:51:30.478 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:30.478 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:51:30.478 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:51:30.478 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:51:30.478 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:51:30.478 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:51:30.478 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:30.481+0000 7f5656ffd640 4 set_mon_vals no callback set 2024-03-29T10:51:30.478 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:30.481+0000 7f565edcf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:51:30.478 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:30.482+0000 7f565edcf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:51:30.478 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:30.482+0000 7f565edcf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-03-29T10:51:30.479 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:30.482+0000 7f565edcf640 1 -- 172.21.15.88:0/71405952 --> v1:172.21.15.88:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5638005080 con 0x7f56580b4920 2024-03-29T10:51:30.479 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:30.483+0000 7f56557fa640 1 -- 172.21.15.88:0/71405952 <== mon.2 v1:172.21.15.88:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4235629220 0 0) 0x7f5648004d50 con 0x7f56580b4920 2024-03-29T10:51:30.479 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:30.483+0000 7f565cb6b640 10 monclient: get_auth_request con 0x7f564406ddb0 auth_method 0 2024-03-29T10:51:30.479 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:30.484+0000 7f56557fa640 1 -- 172.21.15.88:0/71405952 <== mon.2 v1:172.21.15.88:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1407902845 0 0) 0x7f5648084ac0 con 0x7f56580b4920 2024-03-29T10:51:30.479 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:30.484+0000 7f56557fa640 10 client.4664.objecter ms_dispatch 0x7f5658000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T10:51:30.479 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:30.484+0000 7f56557fa640 3 client.4664.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T10:51:30.479 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:30.484+0000 7f56557fa640 3 client.4664.objecter handle_osd_map decoding full epoch 11 2024-03-29T10:51:30.479 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:30.484+0000 7f56557fa640 20 client.4664.objecter dump_active .. 0 homeless 2024-03-29T10:51:30.479 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:30.484+0000 7f56557fa640 10 client.4664.objecter ms_handle_connect 0x7f564406ddb0 2024-03-29T10:51:30.480 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:30.495+0000 7f56557fa640 1 -- 172.21.15.88:0/71405952 <== mon.2 v1:172.21.15.88:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5648050400 con 0x7f56580b4920 2024-03-29T10:51:30.480 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:30.495+0000 7f56557fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:51:30.480 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:30.495+0000 7f56557fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:51:30.783 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:30.802+0000 7f565edcf640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:51:30.784 INFO:teuthology.misc.health.smithi088.stdout: 2024-03-29T10:51:30.785 INFO:teuthology.misc.health.smithi088.stdout:{"epoch":11,"fsid":"869f5188-f92b-42dd-a46d-8c646e0388a9","created":"2024-03-29T10:40:40.285087+0000","modified":"2024-03-29T10:50:45.236723+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:50:45.236723+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a26c0c1c-8de0-438c-a3f6-3df10ea61574","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"5de24915-8dec-49b6-8b16-f3170197c591","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"ac054af3-d589-4565-9644-e6af75975c53","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"3ec4f045-c714-4a22-bd1a-e07330b45ab3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"cddb9867-8e30-4951-9b13-3d1e78b9c45a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dc6d9911-785d-432f-8b53-56a1ccab302f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"77c88437-b587-496e-8b4d-1824ea8b5305","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ce963d96-5a8e-44c3-99de-c0dbebe234ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:51:30.827 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:30.802+0000 7f565edcf640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:51:30.827 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:30.802+0000 7f565edcf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-03-29T10:51:30.827 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:30.802+0000 7f565edcf640 1 -- 172.21.15.88:0/71405952 --> v1:172.21.15.88:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5638005330 con 0x7f56580b4920 2024-03-29T10:51:30.827 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:30.803+0000 7f56557fa640 1 -- 172.21.15.88:0/71405952 <== mon.2 v1:172.21.15.88:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 1123663608) 0x7f5648084170 con 0x7f56580b4920 2024-03-29T10:51:30.827 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:30.803+0000 7f56557fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:51:30.828 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:30.803+0000 7f56557fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:51:30.828 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:30.806+0000 7f565edcf640 10 librados: watch_flush enter 2024-03-29T10:51:30.828 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:30.807+0000 7f565edcf640 10 librados: watch_flush exit 2024-03-29T10:51:30.828 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:30.807+0000 7f565edcf640 20 client.4664.objecter shutdown clearing up homeless session... 2024-03-29T10:51:30.828 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:30.807+0000 7f565edcf640 10 client.4664.objecter successfully canceled tick 2024-03-29T10:51:30.828 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:30.807+0000 7f565edcf640 1 -- 172.21.15.88:0/71405952 >> v1:172.21.15.88:6816/1752861349 conn(0x7f564406ddb0 legacy=0x7f5644070270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:30.828 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:30.807+0000 7f565edcf640 10 monclient: shutdown 2024-03-29T10:51:30.828 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:30.807+0000 7f565edcf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:51:30.828 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:30.807+0000 7f565edcf640 1 -- 172.21.15.88:0/71405952 >> v1:172.21.15.88:6790/0 conn(0x7f56580b4920 legacy=0x7f56581e30d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:30.828 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:30.807+0000 7f565edcf640 1 -- 172.21.15.88:0/71405952 shutdown_connections 2024-03-29T10:51:30.828 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:30.807+0000 7f565edcf640 1 -- 172.21.15.88:0/71405952 >> 172.21.15.88:0/71405952 conn(0x7f5658066a00 msgr2=0x7f5658146c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:51:30.828 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:30.808+0000 7f565edcf640 1 -- 172.21.15.88:0/71405952 shutdown_connections 2024-03-29T10:51:30.828 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:30.808+0000 7f565edcf640 1 -- 172.21.15.88:0/71405952 wait complete. 2024-03-29T10:51:30.828 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:30.808+0000 7f565edcf640 1 librados: shutdown 2024-03-29T10:51:30.842 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:51:37.649 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:51:37.799 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:37.818+0000 7f5dfa34d640 10 monclient: get_monmap_and_config 2024-03-29T10:51:37.799 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:37.818+0000 7f5dfa34d640 10 monclient: build_initial_monmap 2024-03-29T10:51:37.800 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:37.818+0000 7f5dfa34d640 10 monclient: monmap: 2024-03-29T10:51:37.800 INFO:teuthology.misc.health.smithi088.stderr:epoch 0 2024-03-29T10:51:37.800 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:51:37.800 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:51:37.819218+0000 2024-03-29T10:51:37.800 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:51:37.819218+0000 2024-03-29T10:51:37.800 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 0 (unknown) 2024-03-29T10:51:37.800 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:51:37.800 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-03-29T10:51:37.800 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-29T10:51:37.800 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-b 2024-03-29T10:51:37.800 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:51:37.801 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:37.819+0000 7f5dfa34d640 1 Processor -- start 2024-03-29T10:51:37.801 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:37.819+0000 7f5dfa34d640 1 -- start start 2024-03-29T10:51:37.801 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:37.819+0000 7f5dfa34d640 10 monclient: init 2024-03-29T10:51:37.802 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:37.820+0000 7f5dfa34d640 10 monclient: _reopen_session rank -1 2024-03-29T10:51:37.802 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:37.820+0000 7f5dfa34d640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:51:37.802 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:37.820+0000 7f5dfa34d640 10 monclient(hunting): picked mon.noname-b con 0x7f5df41457f0 addr v1:172.21.15.88:6790/0 2024-03-29T10:51:37.802 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:37.820+0000 7f5dfa34d640 10 monclient(hunting): picked mon.noname-a con 0x7f5df413f650 addr v1:172.21.15.88:6789/0 2024-03-29T10:51:37.802 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:37.820+0000 7f5dfa34d640 10 monclient(hunting): picked mon.noname-c con 0x7f5df40a9d70 addr v1:172.21.15.196:6789/0 2024-03-29T10:51:37.802 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:37.820+0000 7f5dfa34d640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5df40aaef0 con 0x7f5df413f650 2024-03-29T10:51:37.802 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:37.820+0000 7f5dfa34d640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5df4145bd0 con 0x7f5df40a9d70 2024-03-29T10:51:37.802 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:37.820+0000 7f5dfa34d640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5df4145eb0 con 0x7f5df41457f0 2024-03-29T10:51:37.802 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:37.820+0000 7f5dfa34d640 10 monclient(hunting): _renew_subs 2024-03-29T10:51:37.803 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:37.820+0000 7f5dfa34d640 10 monclient(hunting): authenticate will time out at 1420.056164s 2024-03-29T10:51:37.803 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:37.820+0000 7f5df37fe640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f5df413f650 0x7f5df40b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:51500/0 (socket says 172.21.15.88:51500) 2024-03-29T10:51:37.803 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:37.820+0000 7f5df37fe640 1 -- 172.21.15.88:0/3548843825 learned_addr learned my addr 172.21.15.88:0/3548843825 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-29T10:51:37.808 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:37.821+0000 7f5df2ffd640 1 -- 172.21.15.88:0/3548843825 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1345252246 0 0) 0x7f5df4145eb0 con 0x7f5df41457f0 2024-03-29T10:51:37.808 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:37.821+0000 7f5df2ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:51:37.808 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:37.821+0000 7f5df2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:37.808 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:37.821+0000 7f5df2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:37.808 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:37.821+0000 7f5df2ffd640 10 monclient(hunting): my global_id is 4667 2024-03-29T10:51:37.808 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:37.821+0000 7f5df2ffd640 1 -- 172.21.15.88:0/3548843825 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ddc003520 con 0x7f5df41457f0 2024-03-29T10:51:37.809 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:37.821+0000 7f5df2ffd640 1 -- 172.21.15.88:0/3548843825 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1578327181 0 0) 0x7f5df4145bd0 con 0x7f5df40a9d70 2024-03-29T10:51:37.809 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:37.821+0000 7f5df2ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:51:37.809 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:37.821+0000 7f5df2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:37.809 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:37.821+0000 7f5df2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:37.809 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:37.821+0000 7f5df2ffd640 10 monclient(hunting): my global_id is 4594 2024-03-29T10:51:37.809 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:37.821+0000 7f5df2ffd640 1 -- 172.21.15.88:0/3548843825 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5df4145eb0 con 0x7f5df40a9d70 2024-03-29T10:51:37.809 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:37.822+0000 7f5df2ffd640 1 -- 172.21.15.88:0/3548843825 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 469117514 0 0) 0x7f5df40aaef0 con 0x7f5df413f650 2024-03-29T10:51:37.809 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:37.822+0000 7f5df2ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:51:37.809 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:37.822+0000 7f5df2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:37.809 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:37.822+0000 7f5df2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:37.809 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:37.822+0000 7f5df2ffd640 10 monclient(hunting): my global_id is 4617 2024-03-29T10:51:37.810 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:37.822+0000 7f5df2ffd640 1 -- 172.21.15.88:0/3548843825 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5df4145bd0 con 0x7f5df413f650 2024-03-29T10:51:37.810 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:37.822+0000 7f5df2ffd640 1 -- 172.21.15.88:0/3548843825 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3888029467 0 0) 0x7f5ddc003520 con 0x7f5df41457f0 2024-03-29T10:51:37.810 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:37.822+0000 7f5df2ffd640 1 -- 172.21.15.88:0/3548843825 >> v1:172.21.15.196:6789/0 conn(0x7f5df40a9d70 legacy=0x7f5df414d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:37.810 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:37.822+0000 7f5df2ffd640 1 -- 172.21.15.88:0/3548843825 >> v1:172.21.15.88:6789/0 conn(0x7f5df413f650 legacy=0x7f5df40b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:37.810 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:37.822+0000 7f5df2ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:51:37.810 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:37.822+0000 7f5df2ffd640 1 monclient: found mon.noname-b 2024-03-29T10:51:37.810 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:37.822+0000 7f5df2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:51:37.810 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:37.822+0000 7f5df2ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.88:6790/0 2024-03-29T10:51:37.810 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:37.822+0000 7f5df2ffd640 1 -- 172.21.15.88:0/3548843825 --> v1:172.21.15.88:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5df4146340 con 0x7f5df41457f0 2024-03-29T10:51:37.810 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:37.822+0000 7f5df2ffd640 10 monclient: _finish_auth 0 2024-03-29T10:51:37.810 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:37.822+0000 7f5df2ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:51:37.811 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:37.822+0000 7f5df2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:51:37.811 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:37.822+0000 7f5df2ffd640 1 -- 172.21.15.88:0/3548843825 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f5dec001990 con 0x7f5df41457f0 2024-03-29T10:51:37.811 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:37.822+0000 7f5df2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:37.811 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:37.822+0000 7f5df2ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:51:37.811 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:37.822+0000 7f5df2ffd640 10 monclient: dump: 2024-03-29T10:51:37.811 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:51:37.811 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:51:37.811 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:51:37.811 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:51:37.811 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:37.811 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:51:37.812 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:51:37.812 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:51:37.812 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:51:37.812 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:51:37.812 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:37.822+0000 7f5dfa34d640 5 monclient: authenticate success, global_id 4667 2024-03-29T10:51:37.812 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:37.822+0000 7f5dfa34d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:51:37.812 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:37.823+0000 7f5df2ffd640 1 -- 172.21.15.88:0/3548843825 <== mon.2 v1:172.21.15.88:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5dec001af0 con 0x7f5df41457f0 2024-03-29T10:51:37.812 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:37.823+0000 7f5df2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:51:37.812 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:37.823+0000 7f5df2ffd640 1 -- 172.21.15.88:0/3548843825 <== mon.2 v1:172.21.15.88:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f5dec001f40 con 0x7f5df41457f0 2024-03-29T10:51:37.812 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:37.823+0000 7f5dfa34d640 10 monclient: get_monmap_and_config success 2024-03-29T10:51:37.812 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:37.823+0000 7f5dfa34d640 4 set_mon_vals no callback set 2024-03-29T10:51:37.812 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:37.823+0000 7f5df2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:37.813 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:37.823+0000 7f5df2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:51:37.813 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:37.823+0000 7f5df2ffd640 10 monclient: dump: 2024-03-29T10:51:37.813 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:51:37.813 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:51:37.813 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:51:37.813 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:51:37.813 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:37.813 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:51:37.813 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:51:37.813 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:51:37.813 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:51:37.813 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:51:37.813 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:37.823+0000 7f5dfa34d640 10 monclient: shutdown 2024-03-29T10:51:37.813 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:37.823+0000 7f5dfa34d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:51:37.813 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:37.823+0000 7f5dfa34d640 1 -- 172.21.15.88:0/3548843825 >> v1:172.21.15.88:6790/0 conn(0x7f5df41457f0 legacy=0x7f5df413ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:37.814 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:37.823+0000 7f5dfa34d640 1 -- 172.21.15.88:0/3548843825 shutdown_connections 2024-03-29T10:51:37.814 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:37.823+0000 7f5dfa34d640 1 -- 172.21.15.88:0/3548843825 >> 172.21.15.88:0/3548843825 conn(0x7f5df4066a00 msgr2=0x7f5df4066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:51:37.814 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:37.824+0000 7f5dfa34d640 1 -- 172.21.15.88:0/3548843825 shutdown_connections 2024-03-29T10:51:37.814 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:37.824+0000 7f5dfa34d640 1 -- 172.21.15.88:0/3548843825 wait complete. 2024-03-29T10:51:37.814 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:37.825+0000 7f5dfa34d640 10 monclient: build_initial_monmap 2024-03-29T10:51:37.814 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:37.825+0000 7f5dfa34d640 10 monclient: monmap: 2024-03-29T10:51:37.814 INFO:teuthology.misc.health.smithi088.stderr:epoch 0 2024-03-29T10:51:37.814 INFO:teuthology.misc.health.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:51:37.814 INFO:teuthology.misc.health.smithi088.stderr:last_changed 0.000000 2024-03-29T10:51:37.814 INFO:teuthology.misc.health.smithi088.stderr:created 0.000000 2024-03-29T10:51:37.814 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 0 (unknown) 2024-03-29T10:51:37.814 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:51:37.814 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-03-29T10:51:37.814 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-29T10:51:37.815 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-c 2024-03-29T10:51:37.815 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:51:37.815 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:37.825+0000 7f5dfa34d640 1 librados: starting msgr at 2024-03-29T10:51:37.815 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:37.825+0000 7f5dfa34d640 1 librados: starting objecter 2024-03-29T10:51:37.815 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:37.825+0000 7f5dfa34d640 1 Processor -- start 2024-03-29T10:51:37.815 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:37.826+0000 7f5dfa34d640 1 -- start start 2024-03-29T10:51:37.815 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:37.826+0000 7f5dfa34d640 1 librados: setting wanted keys 2024-03-29T10:51:37.815 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:37.826+0000 7f5dfa34d640 1 librados: calling monclient init 2024-03-29T10:51:37.815 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:37.826+0000 7f5dfa34d640 10 monclient: init 2024-03-29T10:51:37.823 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:37.827+0000 7f5dfa34d640 10 monclient: _reopen_session rank -1 2024-03-29T10:51:37.823 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:37.827+0000 7f5dfa34d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:51:37.823 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:37.827+0000 7f5dfa34d640 10 monclient(hunting): picked mon.noname-a con 0x7f5df40a9d70 addr v1:172.21.15.88:6789/0 2024-03-29T10:51:37.823 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:37.827+0000 7f5dfa34d640 10 monclient(hunting): picked mon.noname-b con 0x7f5df413f650 addr v1:172.21.15.196:6789/0 2024-03-29T10:51:37.823 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:37.827+0000 7f5dfa34d640 10 monclient(hunting): picked mon.noname-c con 0x7f5df41457f0 addr v1:172.21.15.88:6790/0 2024-03-29T10:51:37.823 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:37.827+0000 7f5dfa34d640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5df41d7bf0 con 0x7f5df40a9d70 2024-03-29T10:51:37.823 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:37.827+0000 7f5dfa34d640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5df41ced90 con 0x7f5df413f650 2024-03-29T10:51:37.823 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:37.827+0000 7f5dfa34d640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5df41d7680 con 0x7f5df41457f0 2024-03-29T10:51:37.823 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:37.827+0000 7f5dfa34d640 10 monclient(hunting): _renew_subs 2024-03-29T10:51:37.823 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:37.827+0000 7f5dfa34d640 10 monclient(hunting): authenticate will time out at 1420.063537s 2024-03-29T10:51:37.824 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:37.827+0000 7f5df3fff640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f5df40a9d70 0x7f5df41cb340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:52118/0 (socket says 172.21.15.88:52118) 2024-03-29T10:51:37.824 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:37.828+0000 7f5df37fe640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f5df413f650 0x7f5df40b0ac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.88:42234/0 (socket says 172.21.15.88:42234) 2024-03-29T10:51:37.824 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:37.828+0000 7f5df37fe640 1 -- 172.21.15.88:0/3836603729 learned_addr learned my addr 172.21.15.88:0/3836603729 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-03-29T10:51:37.824 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:37.828+0000 7f5df0ff9640 10 client.?.objecter ms_handle_connect 0x7f5df413f650 2024-03-29T10:51:37.824 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:37.828+0000 7f5df0ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:51:37.824 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:37.828+0000 7f5df0ff9640 1 -- 172.21.15.88:0/3836603729 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 86526023 0 0) 0x7f5df41ced90 con 0x7f5df413f650 2024-03-29T10:51:37.824 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:37.828+0000 7f5df0ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:51:37.824 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:37.828+0000 7f5df0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:37.824 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:37.828+0000 7f5df0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:37.824 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:37.828+0000 7f5df0ff9640 10 monclient(hunting): my global_id is 4597 2024-03-29T10:51:37.824 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:37.828+0000 7f5df0ff9640 1 -- 172.21.15.88:0/3836603729 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5de0004eb0 con 0x7f5df413f650 2024-03-29T10:51:37.824 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:37.828+0000 7f5df0ff9640 10 client.?.objecter ms_handle_connect 0x7f5df41457f0 2024-03-29T10:51:37.824 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:37.828+0000 7f5df0ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:51:37.824 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:37.828+0000 7f5df0ff9640 10 client.?.objecter ms_handle_connect 0x7f5df40a9d70 2024-03-29T10:51:37.825 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:37.828+0000 7f5df0ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:51:37.825 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:37.829+0000 7f5df0ff9640 1 -- 172.21.15.88:0/3836603729 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2216432348 0 0) 0x7f5df41d7680 con 0x7f5df41457f0 2024-03-29T10:51:37.825 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:37.829+0000 7f5df0ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:51:37.825 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:37.829+0000 7f5df0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:37.825 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:37.829+0000 7f5df0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:37.825 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:37.829+0000 7f5df0ff9640 10 monclient(hunting): my global_id is 4670 2024-03-29T10:51:37.825 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:37.829+0000 7f5df0ff9640 1 -- 172.21.15.88:0/3836603729 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5df41ced90 con 0x7f5df41457f0 2024-03-29T10:51:37.825 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:37.829+0000 7f5df0ff9640 1 -- 172.21.15.88:0/3836603729 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4136496626 0 0) 0x7f5df41d7bf0 con 0x7f5df40a9d70 2024-03-29T10:51:37.825 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:37.829+0000 7f5df0ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:51:37.825 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:37.829+0000 7f5df0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:37.825 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:37.829+0000 7f5df0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:37.825 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:37.829+0000 7f5df0ff9640 10 monclient(hunting): my global_id is 4620 2024-03-29T10:51:37.825 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:37.829+0000 7f5df0ff9640 1 -- 172.21.15.88:0/3836603729 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5df41d7680 con 0x7f5df40a9d70 2024-03-29T10:51:37.825 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:37.829+0000 7f5df0ff9640 1 -- 172.21.15.88:0/3836603729 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3093924141 0 0) 0x7f5df41ced90 con 0x7f5df41457f0 2024-03-29T10:51:37.825 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:37.829+0000 7f5df0ff9640 1 -- 172.21.15.88:0/3836603729 --> v1:172.21.15.88:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5df41d7bf0 con 0x7f5df41457f0 2024-03-29T10:51:37.826 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:37.829+0000 7f5df0ff9640 1 -- 172.21.15.88:0/3836603729 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f5de4001930 con 0x7f5df41457f0 2024-03-29T10:51:37.826 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:37.829+0000 7f5df0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:37.826 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:37.829+0000 7f5df0ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:51:37.826 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:37.829+0000 7f5df0ff9640 10 monclient(hunting): dump: 2024-03-29T10:51:37.826 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:51:37.826 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:51:37.826 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:51:37.826 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:51:37.826 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:37.826 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:51:37.826 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:51:37.826 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:51:37.826 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:51:37.826 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:51:37.827 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:37.829+0000 7f5df0ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:51:37.827 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:37.829+0000 7f5df0ff9640 1 -- 172.21.15.88:0/3836603729 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1768151683 0 0) 0x7f5de0004eb0 con 0x7f5df413f650 2024-03-29T10:51:37.827 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:37.829+0000 7f5df0ff9640 1 -- 172.21.15.88:0/3836603729 --> v1:172.21.15.196:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5df41ced90 con 0x7f5df413f650 2024-03-29T10:51:37.827 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:37.829+0000 7f5df0ff9640 1 -- 172.21.15.88:0/3836603729 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f5de8001950 con 0x7f5df413f650 2024-03-29T10:51:37.827 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:37.829+0000 7f5df0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:37.827 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:37.829+0000 7f5df0ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T10:51:37.827 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:37.829+0000 7f5df0ff9640 10 monclient(hunting): dump: 2024-03-29T10:51:37.827 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:51:37.827 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:51:37.827 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:51:37.827 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:51:37.827 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:37.827 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:51:37.827 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:51:37.828 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:51:37.828 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:51:37.828 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:51:37.828 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:37.829+0000 7f5df0ff9640 1 -- 172.21.15.88:0/3836603729 <== mon.2 v1:172.21.15.88:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3801500054 0 0) 0x7f5df41d7bf0 con 0x7f5df41457f0 2024-03-29T10:51:37.828 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:37.829+0000 7f5df0ff9640 1 -- 172.21.15.88:0/3836603729 >> v1:172.21.15.196:6789/0 conn(0x7f5df413f650 legacy=0x7f5df40b0ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:37.828 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:37.829+0000 7f5df0ff9640 1 -- 172.21.15.88:0/3836603729 >> v1:172.21.15.88:6789/0 conn(0x7f5df40a9d70 legacy=0x7f5df41cb340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:37.828 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:37.829+0000 7f5df0ff9640 10 monclient: _finish_hunting 0 2024-03-29T10:51:37.828 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:37.829+0000 7f5df0ff9640 1 monclient: found mon.c 2024-03-29T10:51:37.828 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:37.829+0000 7f5df0ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:51:37.828 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:37.829+0000 7f5df0ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-03-29T10:51:37.828 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:37.829+0000 7f5df0ff9640 1 -- 172.21.15.88:0/3836603729 --> v1:172.21.15.88:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5df41d7810 con 0x7f5df41457f0 2024-03-29T10:51:37.828 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:37.830+0000 7f5df0ff9640 10 monclient: _finish_auth 0 2024-03-29T10:51:37.828 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:37.830+0000 7f5df0ff9640 10 monclient: _check_auth_tickets 2024-03-29T10:51:37.828 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:37.830+0000 7f5df0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:51:37.829 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:37.830+0000 7f5dfa34d640 5 monclient: authenticate success, global_id 4670 2024-03-29T10:51:37.829 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:37.830+0000 7f5dfa34d640 10 monclient: _renew_subs 2024-03-29T10:51:37.829 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:37.830+0000 7f5dfa34d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-03-29T10:51:37.829 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:37.830+0000 7f5dfa34d640 1 -- 172.21.15.88:0/3836603729 --> v1:172.21.15.88:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5df4139bc0 con 0x7f5df41457f0 2024-03-29T10:51:37.829 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:37.830+0000 7f5dfa34d640 10 client.4670.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:51:37.829 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:37.830+0000 7f5dfa34d640 10 monclient: _renew_subs 2024-03-29T10:51:37.829 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:37.830+0000 7f5dfa34d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-03-29T10:51:37.829 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:37.830+0000 7f5dfa34d640 1 -- 172.21.15.88:0/3836603729 --> v1:172.21.15.88:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5df4141fe0 con 0x7f5df41457f0 2024-03-29T10:51:37.829 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:37.830+0000 7f5dfa34d640 1 librados: init done 2024-03-29T10:51:37.829 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:37.831+0000 7f5dfa34d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:51:37.830 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:37.831+0000 7f5dfa34d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:51:37.830 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:37.831+0000 7f5dfa34d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-03-29T10:51:37.830 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:37.831+0000 7f5dfa34d640 1 -- 172.21.15.88:0/3836603729 --> v1:172.21.15.88:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5df4075f10 con 0x7f5df41457f0 2024-03-29T10:51:37.830 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:37.832+0000 7f5df0ff9640 1 -- 172.21.15.88:0/3836603729 <== mon.2 v1:172.21.15.88:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5de4003610 con 0x7f5df41457f0 2024-03-29T10:51:37.830 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:37.832+0000 7f5df0ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:51:37.830 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:37.832+0000 7f5df0ff9640 1 -- 172.21.15.88:0/3836603729 <== mon.2 v1:172.21.15.88:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1853843833 0 0) 0x7f5de40039a0 con 0x7f5df41457f0 2024-03-29T10:51:37.830 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:37.832+0000 7f5df0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:37.830 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:37.832+0000 7f5df0ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:51:37.830 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:37.832+0000 7f5df0ff9640 10 monclient: dump: 2024-03-29T10:51:37.831 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-03-29T10:51:37.831 INFO:teuthology.misc.health.smithi088.stderr:fsid 869f5188-f92b-42dd-a46d-8c646e0388a9 2024-03-29T10:51:37.831 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-03-29T10:40:02.019250+0000 2024-03-29T10:51:37.831 INFO:teuthology.misc.health.smithi088.stderr:created 2024-03-29T10:40:02.019250+0000 2024-03-29T10:51:37.831 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:37.831 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-03-29T10:51:37.831 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-03-29T10:51:37.832 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T10:51:37.832 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-03-29T10:51:37.832 INFO:teuthology.misc.health.smithi088.stderr: 2024-03-29T10:51:37.832 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:37.832+0000 7f5df27fc640 4 set_mon_vals no callback set 2024-03-29T10:51:37.832 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:37.833+0000 7f5df0ff9640 1 -- 172.21.15.88:0/3836603729 <== mon.2 v1:172.21.15.88:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4235629220 0 0) 0x7f5de4003bc0 con 0x7f5df41457f0 2024-03-29T10:51:37.832 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:37.833+0000 7f5df0ff9640 1 -- 172.21.15.88:0/3836603729 <== mon.2 v1:172.21.15.88:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1407902845 0 0) 0x7f5de40835e0 con 0x7f5df41457f0 2024-03-29T10:51:37.832 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:37.833+0000 7f5df0ff9640 10 client.4670.objecter ms_dispatch 0x7f5df4000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T10:51:37.832 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:37.833+0000 7f5df0ff9640 3 client.4670.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T10:51:37.833 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:37.833+0000 7f5df0ff9640 3 client.4670.objecter handle_osd_map decoding full epoch 11 2024-03-29T10:51:37.833 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:37.834+0000 7f5df0ff9640 20 client.4670.objecter dump_active .. 0 homeless 2024-03-29T10:51:37.833 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:37.834+0000 7f5df3fff640 10 monclient: get_auth_request con 0x7f5de006df80 auth_method 0 2024-03-29T10:51:37.833 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:37.834+0000 7f5df0ff9640 10 client.4670.objecter ms_handle_connect 0x7f5de006df80 2024-03-29T10:51:37.833 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:37.838+0000 7f5df0ff9640 1 -- 172.21.15.88:0/3836603729 <== mon.2 v1:172.21.15.88:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5de404ef20 con 0x7f5df41457f0 2024-03-29T10:51:37.833 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:37.838+0000 7f5df0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:51:37.833 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:37.838+0000 7f5df0ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:51:38.178 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:38.196+0000 7f5dfa34d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:51:38.178 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:38.196+0000 7f5dfa34d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:51:38.178 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:38.196+0000 7f5dfa34d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-03-29T10:51:38.178 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:38.196+0000 7f5dfa34d640 1 -- 172.21.15.88:0/3836603729 --> v1:172.21.15.88:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5df41d79a0 con 0x7f5df41457f0 2024-03-29T10:51:38.179 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:38.198+0000 7f5df0ff9640 1 -- 172.21.15.88:0/3836603729 <== mon.2 v1:172.21.15.88:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 1123663608) 0x7f5de4050e50 con 0x7f5df41457f0 2024-03-29T10:51:38.179 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:38.198+0000 7f5df0ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:51:38.179 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:38.198+0000 7f5df0ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:51:38.180 INFO:teuthology.misc.health.smithi088.stdout: 2024-03-29T10:51:38.180 INFO:teuthology.misc.health.smithi088.stdout:{"epoch":11,"fsid":"869f5188-f92b-42dd-a46d-8c646e0388a9","created":"2024-03-29T10:40:40.285087+0000","modified":"2024-03-29T10:50:45.236723+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:50:45.236723+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a26c0c1c-8de0-438c-a3f6-3df10ea61574","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"5de24915-8dec-49b6-8b16-f3170197c591","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"ac054af3-d589-4565-9644-e6af75975c53","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"3ec4f045-c714-4a22-bd1a-e07330b45ab3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"cddb9867-8e30-4951-9b13-3d1e78b9c45a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dc6d9911-785d-432f-8b53-56a1ccab302f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"77c88437-b587-496e-8b4d-1824ea8b5305","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ce963d96-5a8e-44c3-99de-c0dbebe234ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:51:38.184 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:38.202+0000 7f5dfa34d640 10 librados: watch_flush enter 2024-03-29T10:51:38.185 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:38.202+0000 7f5dfa34d640 10 librados: watch_flush exit 2024-03-29T10:51:38.185 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:38.202+0000 7f5dfa34d640 20 client.4670.objecter shutdown clearing up homeless session... 2024-03-29T10:51:38.185 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:38.202+0000 7f5dfa34d640 10 client.4670.objecter successfully canceled tick 2024-03-29T10:51:38.185 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:38.202+0000 7f5dfa34d640 1 -- 172.21.15.88:0/3836603729 >> v1:172.21.15.88:6816/1752861349 conn(0x7f5de006df80 legacy=0x7f5de0070440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:38.185 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:38.202+0000 7f5dfa34d640 10 monclient: shutdown 2024-03-29T10:51:38.185 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:38.202+0000 7f5dfa34d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:51:38.185 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:38.202+0000 7f5dfa34d640 1 -- 172.21.15.88:0/3836603729 >> v1:172.21.15.88:6790/0 conn(0x7f5df41457f0 legacy=0x7f5df41e2ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:38.185 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:38.203+0000 7f5dfa34d640 1 -- 172.21.15.88:0/3836603729 shutdown_connections 2024-03-29T10:51:38.185 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:38.203+0000 7f5dfa34d640 1 -- 172.21.15.88:0/3836603729 >> 172.21.15.88:0/3836603729 conn(0x7f5df4066a00 msgr2=0x7f5df413ab40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:51:38.185 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:38.203+0000 7f5dfa34d640 1 -- 172.21.15.88:0/3836603729 shutdown_connections 2024-03-29T10:51:38.185 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:38.203+0000 7f5dfa34d640 1 -- 172.21.15.88:0/3836603729 wait complete. 2024-03-29T10:51:38.185 INFO:teuthology.misc.health.smithi088.stderr:2024-03-29T10:51:38.203+0000 7f5dfa34d640 1 librados: shutdown 2024-03-29T10:51:38.200 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:51:38.200 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds 2024-03-29T10:51:38.202 INFO:teuthology.misc:Shutting down osd daemons... 2024-03-29T10:51:38.203 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-03-29T10:51:38.203 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T10:51:38.235 INFO:tasks.ceph.osd.0:Stopped 2024-03-29T10:51:38.235 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-03-29T10:51:38.235 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T10:51:38.305 INFO:tasks.ceph.osd.1:Stopped 2024-03-29T10:51:38.305 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-03-29T10:51:38.305 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T10:51:38.383 INFO:tasks.ceph.osd.2:Stopped 2024-03-29T10:51:38.384 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-03-29T10:51:38.384 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T10:51:38.464 INFO:tasks.ceph.osd.3:Stopped 2024-03-29T10:51:38.464 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-03-29T10:51:38.464 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T10:51:38.503 INFO:tasks.ceph.osd.4:Stopped 2024-03-29T10:51:38.503 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-03-29T10:51:38.503 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T10:51:38.574 INFO:tasks.ceph.osd.5:Stopped 2024-03-29T10:51:38.574 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-03-29T10:51:38.574 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T10:51:38.645 INFO:tasks.ceph.osd.6:Stopped 2024-03-29T10:51:38.646 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-03-29T10:51:38.646 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T10:51:38.715 INFO:tasks.ceph.osd.7:Stopped 2024-03-29T10:51:38.715 INFO:teuthology.misc:Shutting down mgr daemons... 2024-03-29T10:51:38.715 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-03-29T10:51:38.715 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T10:51:38.739 INFO:tasks.ceph.mgr.y:Stopped 2024-03-29T10:51:38.740 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-03-29T10:51:38.740 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T10:51:38.780 INFO:tasks.ceph.mgr.x:Stopped 2024-03-29T10:51:38.780 INFO:teuthology.misc:Shutting down mon daemons... 2024-03-29T10:51:38.780 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-03-29T10:51:38.780 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T10:51:38.792 INFO:tasks.ceph.mon.a:Stopped 2024-03-29T10:51:38.792 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-03-29T10:51:38.792 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T10:51:38.843 INFO:tasks.ceph.mon.c:Stopped 2024-03-29T10:51:38.844 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-03-29T10:51:38.844 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T10:51:38.854 INFO:tasks.ceph.mon.b:Stopped 2024-03-29T10:51:38.855 INFO:tasks.ceph:Checking cluster log for badness... 2024-03-29T10:51:38.855 DEBUG:teuthology.orchestra.run.smithi088:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'reached quota' | egrep -v 'overall HEALTH_' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(POOL_FULL\)' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(CACHE_POOL_NEAR_FULL\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(OBJECT_MISPLACED\)' | egrep -v 'slow request' | head -n 1 2024-03-29T10:51:38.916 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi088.front.sepia.ceph.com 2024-03-29T10:51:38.917 DEBUG:teuthology.orchestra.run.smithi088:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-03-29T10:51:41.951 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi088.front.sepia.ceph.com 2024-03-29T10:51:41.951 DEBUG:teuthology.orchestra.run.smithi088:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-03-29T10:51:42.087 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi088.front.sepia.ceph.com 2024-03-29T10:51:42.087 DEBUG:teuthology.orchestra.run.smithi088:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-03-29T10:51:42.163 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi088.front.sepia.ceph.com 2024-03-29T10:51:42.163 DEBUG:teuthology.orchestra.run.smithi088:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-03-29T10:51:42.311 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi196.front.sepia.ceph.com 2024-03-29T10:51:42.311 DEBUG:teuthology.orchestra.run.smithi196:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-03-29T10:51:45.819 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi196.front.sepia.ceph.com 2024-03-29T10:51:45.820 DEBUG:teuthology.orchestra.run.smithi196:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-03-29T10:51:45.912 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi196.front.sepia.ceph.com 2024-03-29T10:51:45.912 DEBUG:teuthology.orchestra.run.smithi196:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-03-29T10:51:45.979 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi196.front.sepia.ceph.com 2024-03-29T10:51:45.979 DEBUG:teuthology.orchestra.run.smithi196:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-03-29T10:51:46.038 INFO:tasks.ceph:Archiving mon data... 2024-03-29T10:51:46.040 DEBUG:teuthology.misc:Transferring archived files from smithi088:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622648/data/mon.a.tgz 2024-03-29T10:51:46.040 DEBUG:teuthology.orchestra.run.smithi088:> mktemp 2024-03-29T10:51:46.058 INFO:teuthology.orchestra.run.smithi088.stdout:/tmp/tmp.G0QKGDpfXQ 2024-03-29T10:51:46.058 DEBUG:teuthology.orchestra.run.smithi088:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.G0QKGDpfXQ 2024-03-29T10:51:46.167 DEBUG:teuthology.orchestra.run.smithi088:> sudo chmod 0666 /tmp/tmp.G0QKGDpfXQ 2024-03-29T10:51:46.255 DEBUG:teuthology.orchestra.remote:smithi088:/tmp/tmp.G0QKGDpfXQ is 126KB 2024-03-29T10:51:46.307 DEBUG:teuthology.orchestra.run.smithi088:> rm -fr /tmp/tmp.G0QKGDpfXQ 2024-03-29T10:51:46.322 DEBUG:teuthology.misc:Transferring archived files from smithi088:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622648/data/mon.c.tgz 2024-03-29T10:51:46.323 DEBUG:teuthology.orchestra.run.smithi088:> mktemp 2024-03-29T10:51:46.377 INFO:teuthology.orchestra.run.smithi088.stdout:/tmp/tmp.avvhuPOmaT 2024-03-29T10:51:46.377 DEBUG:teuthology.orchestra.run.smithi088:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.avvhuPOmaT 2024-03-29T10:51:46.465 DEBUG:teuthology.orchestra.run.smithi088:> sudo chmod 0666 /tmp/tmp.avvhuPOmaT 2024-03-29T10:51:46.553 DEBUG:teuthology.orchestra.remote:smithi088:/tmp/tmp.avvhuPOmaT is 126KB 2024-03-29T10:51:46.604 DEBUG:teuthology.orchestra.run.smithi088:> rm -fr /tmp/tmp.avvhuPOmaT 2024-03-29T10:51:46.619 DEBUG:teuthology.misc:Transferring archived files from smithi196:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622648/data/mon.b.tgz 2024-03-29T10:51:46.619 DEBUG:teuthology.orchestra.run.smithi196:> mktemp 2024-03-29T10:51:46.636 INFO:teuthology.orchestra.run.smithi196.stdout:/tmp/tmp.DMUT3c2fYA 2024-03-29T10:51:46.637 DEBUG:teuthology.orchestra.run.smithi196:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.DMUT3c2fYA 2024-03-29T10:51:46.740 DEBUG:teuthology.orchestra.run.smithi196:> sudo chmod 0666 /tmp/tmp.DMUT3c2fYA 2024-03-29T10:51:46.828 DEBUG:teuthology.orchestra.remote:smithi196:/tmp/tmp.DMUT3c2fYA is 126KB 2024-03-29T10:51:46.881 DEBUG:teuthology.orchestra.run.smithi196:> rm -fr /tmp/tmp.DMUT3c2fYA 2024-03-29T10:51:46.896 INFO:tasks.ceph:Cleaning ceph cluster... 2024-03-29T10:51:46.896 DEBUG:teuthology.orchestra.run.smithi088:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-29T10:51:46.898 DEBUG:teuthology.orchestra.run.smithi196:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-29T10:51:46.978 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-03-29T10:51:47.019 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-03-29T10:51:47.019 INFO:tasks.ceph:Archiving crash dumps... 2024-03-29T10:51:47.021 DEBUG:teuthology.misc:Transferring archived files from smithi088:/var/lib/ceph/crash to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622648/remote/smithi088/crash 2024-03-29T10:51:47.022 DEBUG:teuthology.orchestra.run.smithi088:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-29T10:51:47.048 DEBUG:teuthology.misc:Transferring archived files from smithi196:/var/lib/ceph/crash to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622648/remote/smithi196/crash 2024-03-29T10:51:47.049 DEBUG:teuthology.orchestra.run.smithi196:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-29T10:51:47.086 INFO:tasks.ceph:Compressing logs... 2024-03-29T10:51:47.087 DEBUG:teuthology.orchestra.run.smithi088:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-29T10:51:47.092 DEBUG:teuthology.orchestra.run.smithi196:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-29T10:51:47.160 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40550.log 2024-03-29T10:51:47.160 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41623.log 2024-03-29T10:51:47.160 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-03-29T10:51:47.160 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.40550.log: /var/log/ceph/ceph-client.admin.41623.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41022.log 2024-03-29T10:51:47.160 INFO:teuthology.orchestra.run.smithi088.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.40550.log.gz 2024-03-29T10:51:47.160 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-mon.a.log: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.41623.log.gz 2024-03-29T10:51:47.160 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39925.log 2024-03-29T10:51:47.160 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.41022.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38911.log 2024-03-29T10:51:47.160 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.39925.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41373.log 2024-03-29T10:51:47.160 INFO:teuthology.orchestra.run.smithi088.stderr: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.41022.log.gz 2024-03-29T10:51:47.160 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.38911.log: 85.4% -- replaced with /var/log/ceph/ceph-client.admin.39925.log.gz 2024-03-29T10:51:47.160 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41584.log 2024-03-29T10:51:47.161 INFO:teuthology.orchestra.run.smithi088.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.38911.log.gzgzip 2024-03-29T10:51:47.161 INFO:teuthology.orchestra.run.smithi088.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.39496.log 2024-03-29T10:51:47.161 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.41373.log: /var/log/ceph/ceph-client.admin.41584.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40393.log 2024-03-29T10:51:47.161 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.39496.log: gzip -5 --verbose 85.7% -- /var/log/ceph/ceph-client.admin.40198.log -- replaced with /var/log/ceph/ceph-client.admin.41373.log.gz 2024-03-29T10:51:47.161 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-03-29T10:51:47.161 INFO:teuthology.orchestra.run.smithi088.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.41584.log.gz 2024-03-29T10:51:47.161 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.40393.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40788.log 2024-03-29T10:51:47.161 INFO:teuthology.orchestra.run.smithi088.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.39496.log.gz 2024-03-29T10:51:47.161 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.40198.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39301.log 2024-03-29T10:51:47.161 INFO:teuthology.orchestra.run.smithi088.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.40393.log.gz 2024-03-29T10:51:47.161 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.40788.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.40198.log.gz 2024-03-29T10:51:47.161 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40745.log 2024-03-29T10:51:47.161 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.39301.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41295.log 2024-03-29T10:51:47.161 INFO:teuthology.orchestra.run.smithi088.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.40788.log.gz 2024-03-29T10:51:47.162 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.40745.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40589.log 2024-03-29T10:51:47.162 INFO:teuthology.orchestra.run.smithi088.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.39301.log.gz 2024-03-29T10:51:47.162 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.41295.log: 85.6% -- replaced with /var/log/ceph/ceph-client.admin.40745.log.gzgzip 2024-03-29T10:51:47.162 INFO:teuthology.orchestra.run.smithi088.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.40042.log 2024-03-29T10:51:47.162 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.40589.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41818.log 2024-03-29T10:51:47.162 INFO:teuthology.orchestra.run.smithi088.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.41295.log.gz 2024-03-29T10:51:47.162 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.40042.log: gzip -5 --verbose -- 85.9% /var/log/ceph/ceph-client.admin.39535.log 2024-03-29T10:51:47.162 INFO:teuthology.orchestra.run.smithi088.stderr: -- replaced with /var/log/ceph/ceph-client.admin.40589.log.gz 2024-03-29T10:51:47.162 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.41818.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40628.log 2024-03-29T10:51:47.162 INFO:teuthology.orchestra.run.smithi088.stderr: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.40042.log.gz 2024-03-29T10:51:47.162 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.39535.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41935.log 2024-03-29T10:51:47.162 INFO:teuthology.orchestra.run.smithi088.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.41818.log.gz 2024-03-29T10:51:47.162 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.40628.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.39535.log.gz 2024-03-29T10:51:47.162 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39691.log 2024-03-29T10:51:47.163 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.41935.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41100.log 2024-03-29T10:51:47.163 INFO:teuthology.orchestra.run.smithi088.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.40628.log.gz 2024-03-29T10:51:47.163 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.39691.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40827.log 2024-03-29T10:51:47.163 INFO:teuthology.orchestra.run.smithi088.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.41935.log.gz 2024-03-29T10:51:47.163 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.41100.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.39691.log.gz 2024-03-29T10:51:47.163 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41334.log 2024-03-29T10:51:47.163 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.40827.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39769.log 2024-03-29T10:51:47.163 INFO:teuthology.orchestra.run.smithi088.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.41100.log.gz 2024-03-29T10:51:47.163 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.41334.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40315.log 2024-03-29T10:51:47.163 INFO:teuthology.orchestra.run.smithi088.stderr: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.40827.log.gz 2024-03-29T10:51:47.163 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.39769.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41217.log 2024-03-29T10:51:47.163 INFO:teuthology.orchestra.run.smithi088.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.41334.log.gz 2024-03-29T10:51:47.163 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.40315.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39652.log 2024-03-29T10:51:47.163 INFO:teuthology.orchestra.run.smithi088.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.39769.log.gz 2024-03-29T10:51:47.163 INFO:teuthology.orchestra.run.smithi088.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.40315.log.gz 2024-03-29T10:51:47.164 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.41217.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38794.log 2024-03-29T10:51:47.164 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.39652.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-29T10:51:47.164 INFO:teuthology.orchestra.run.smithi088.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.41217.log.gz 2024-03-29T10:51:47.164 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.38794.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40706.log 2024-03-29T10:51:47.164 INFO:teuthology.orchestra.run.smithi088.stderr: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.39652.log.gz 2024-03-29T10:51:47.164 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38597.log 2024-03-29T10:51:47.164 INFO:teuthology.orchestra.run.smithi088.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.38794.log.gz 2024-03-29T10:51:47.164 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.40706.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41779.log 2024-03-29T10:51:47.164 INFO:teuthology.orchestra.run.smithi088.stderr: 93.4% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-29T10:51:47.164 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.38597.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39262.log 2024-03-29T10:51:47.164 INFO:teuthology.orchestra.run.smithi088.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.40706.log.gz 2024-03-29T10:51:47.164 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39145.log 2024-03-29T10:51:47.164 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.41779.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.38597.log.gz 2024-03-29T10:51:47.164 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.39262.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40905.log 2024-03-29T10:51:47.164 INFO:teuthology.orchestra.run.smithi088.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.41779.log.gz 2024-03-29T10:51:47.165 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.39145.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41701.log 2024-03-29T10:51:47.165 INFO:teuthology.orchestra.run.smithi088.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.39262.log.gz 2024-03-29T10:51:47.165 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39574.log 2024-03-29T10:51:47.165 INFO:teuthology.orchestra.run.smithi088.stderr: 85.9%/var/log/ceph/ceph-client.admin.40905.log: -- replaced with /var/log/ceph/ceph-client.admin.39145.log.gz 2024-03-29T10:51:47.165 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.41701.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41545.log 2024-03-29T10:51:47.165 INFO:teuthology.orchestra.run.smithi088.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.40905.log.gz 2024-03-29T10:51:47.165 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.39574.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40983.log 2024-03-29T10:51:47.165 INFO:teuthology.orchestra.run.smithi088.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.41701.log.gz 2024-03-29T10:51:47.165 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.41545.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40667.log 2024-03-29T10:51:47.165 INFO:teuthology.orchestra.run.smithi088.stderr: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.39574.log.gz 2024-03-29T10:51:47.165 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.40983.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41256.log 2024-03-29T10:51:47.165 INFO:teuthology.orchestra.run.smithi088.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.41545.log.gz 2024-03-29T10:51:47.165 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.40667.log: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.40983.log.gz 2024-03-29T10:51:47.165 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40159.log 2024-03-29T10:51:47.166 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.41256.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39886.log 2024-03-29T10:51:47.166 INFO:teuthology.orchestra.run.smithi088.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.40667.log.gz 2024-03-29T10:51:47.166 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.40159.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40471.log 2024-03-29T10:51:47.166 INFO:teuthology.orchestra.run.smithi088.stderr: 85.6% -- replaced with /var/log/ceph/ceph-client.admin.41256.log.gz 2024-03-29T10:51:47.166 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.39886.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-03-29T10:51:47.166 INFO:teuthology.orchestra.run.smithi088.stderr: 85.5% -- replaced with /var/log/ceph/ceph-client.admin.40159.log.gz 2024-03-29T10:51:47.166 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.40471.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39808.log 2024-03-29T10:51:47.166 INFO:teuthology.orchestra.run.smithi088.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.39886.log.gz 2024-03-29T10:51:47.166 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41857.log 2024-03-29T10:51:47.166 INFO:teuthology.orchestra.run.smithi088.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.40471.log.gz 2024-03-29T10:51:47.166 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.39808.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39964.log 2024-03-29T10:51:47.166 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.41857.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34789.log 2024-03-29T10:51:47.166 INFO:teuthology.orchestra.run.smithi088.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.39808.log.gz 2024-03-29T10:51:47.166 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.39964.log: 86.3%gzip -- replaced with /var/log/ceph/ceph-client.admin.41857.log.gz -5 2024-03-29T10:51:47.167 INFO:teuthology.orchestra.run.smithi088.stderr: --verbose -- /var/log/ceph/ceph-client.admin.39067.log 2024-03-29T10:51:47.167 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.34789.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.28981.log 2024-03-29T10:51:47.167 INFO:teuthology.orchestra.run.smithi088.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.39964.log.gz 2024-03-29T10:51:47.167 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.39067.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40003.log 2024-03-29T10:51:47.167 INFO:teuthology.orchestra.run.smithi088.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.34789.log.gz 2024-03-29T10:51:47.167 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph.tmp-client.admin.28981.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40120.log 2024-03-29T10:51:47.167 INFO:teuthology.orchestra.run.smithi088.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.39067.log.gz 2024-03-29T10:51:47.167 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.28981.log.gz 2024-03-29T10:51:47.167 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40510.log 2024-03-29T10:51:47.167 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.40003.log: /var/log/ceph/ceph-client.admin.40120.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39223.log 2024-03-29T10:51:47.167 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.40510.log: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.40003.log.gz 2024-03-29T10:51:47.167 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38989.log 2024-03-29T10:51:47.167 INFO:teuthology.orchestra.run.smithi088.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.40120.log.gz 2024-03-29T10:51:47.167 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.39223.log: 85.5% -- replaced with /var/log/ceph/ceph-client.admin.40510.log.gz 2024-03-29T10:51:47.167 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35083.log 2024-03-29T10:51:47.168 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-03-29T10:51:47.168 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.38989.log: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.39223.log.gz 2024-03-29T10:51:47.168 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.35083.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39028.log 2024-03-29T10:51:47.168 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-osd.3.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.38989.log.gz 2024-03-29T10:51:47.168 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41061.log 2024-03-29T10:51:47.168 INFO:teuthology.orchestra.run.smithi088.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.35083.log.gz 2024-03-29T10:51:47.168 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.39028.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40432.log 2024-03-29T10:51:47.168 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.41061.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38519.log 2024-03-29T10:51:47.168 INFO:teuthology.orchestra.run.smithi088.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.39028.log.gz 2024-03-29T10:51:47.168 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.40432.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38714.log 2024-03-29T10:51:47.168 INFO:teuthology.orchestra.run.smithi088.stderr: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.41061.log.gz 2024-03-29T10:51:47.168 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.38519.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40237.log 2024-03-29T10:51:47.168 INFO:teuthology.orchestra.run.smithi088.stderr: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.40432.log.gz 2024-03-29T10:51:47.168 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.38714.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39106.log 2024-03-29T10:51:47.169 INFO:teuthology.orchestra.run.smithi088.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.38519.log.gz 2024-03-29T10:51:47.169 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.40237.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41139.log 2024-03-29T10:51:47.169 INFO:teuthology.orchestra.run.smithi088.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.38714.log.gz 2024-03-29T10:51:47.169 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.39106.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40276.log 2024-03-29T10:51:47.169 INFO:teuthology.orchestra.run.smithi088.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.40237.log.gz 2024-03-29T10:51:47.169 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.41139.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40866.log 2024-03-29T10:51:47.169 INFO:teuthology.orchestra.run.smithi088.stderr: 85.2% -- replaced with /var/log/ceph/ceph-client.admin.39106.log.gz 2024-03-29T10:51:47.169 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.40276.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2024-03-29T10:51:47.169 INFO:teuthology.orchestra.run.smithi088.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.41139.log.gz 2024-03-29T10:51:47.169 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.40866.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39340.log 2024-03-29T10:51:47.169 INFO:teuthology.orchestra.run.smithi088.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.40276.log.gz 2024-03-29T10:51:47.169 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38950.log 2024-03-29T10:51:47.169 INFO:teuthology.orchestra.run.smithi088.stderr: 85.6% -- replaced with /var/log/ceph/ceph-client.admin.40866.log.gz 2024-03-29T10:51:47.169 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.39340.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40944.log 2024-03-29T10:51:47.170 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.38950.log: gzip -5 --verbose -- 85.8% -- replaced with /var/log/ceph/ceph-client.admin.39340.log.gz 2024-03-29T10:51:47.170 INFO:teuthology.orchestra.run.smithi088.stderr: /var/log/ceph/ceph-osd.0.log 2024-03-29T10:51:47.170 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.40944.log: 85.6%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39613.log 2024-03-29T10:51:47.170 INFO:teuthology.orchestra.run.smithi088.stderr: -- replaced with /var/log/ceph/ceph-client.admin.38950.log.gz 2024-03-29T10:51:47.170 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41662.log 2024-03-29T10:51:47.170 INFO:teuthology.orchestra.run.smithi088.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.40944.log.gz 2024-03-29T10:51:47.170 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.39613.log: -- /var/log/ceph/ceph-client.admin.39184.log 2024-03-29T10:51:47.170 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.41662.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39847.log 2024-03-29T10:51:47.170 INFO:teuthology.orchestra.run.smithi088.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.39613.log.gz 2024-03-29T10:51:47.170 INFO:teuthology.orchestra.run.smithi088.stderr: 85.6%/var/log/ceph/ceph-client.admin.39184.log: -- replaced with /var/log/ceph/ceph-client.admin.41662.log.gz 2024-03-29T10:51:47.170 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-03-29T10:51:47.170 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.39847.log: 86.5%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41896.log 2024-03-29T10:51:47.170 INFO:teuthology.orchestra.run.smithi088.stderr: -- replaced with /var/log/ceph/ceph-client.admin.39184.log.gz 2024-03-29T10:51:47.170 INFO:teuthology.orchestra.run.smithi088.stderr: 86.8%/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39418.log 2024-03-29T10:51:47.171 INFO:teuthology.orchestra.run.smithi088.stderr: -- replaced with /var/log/ceph/ceph-client.admin.39847.log.gz 2024-03-29T10:51:47.171 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.41896.log: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.41896.log.gz 2024-03-29T10:51:47.171 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38833.log 2024-03-29T10:51:47.171 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.39418.log: 92.8% -- replaced with /var/log/ceph/ceph.log.gz 2024-03-29T10:51:47.171 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-03-29T10:51:47.171 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.38833.log: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.39418.log.gz 2024-03-29T10:51:47.171 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35183.log 2024-03-29T10:51:47.171 INFO:teuthology.orchestra.run.smithi088.stderr: 86.5%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39379.log 2024-03-29T10:51:47.171 INFO:teuthology.orchestra.run.smithi088.stderr: -- replaced with /var/log/ceph/ceph-client.admin.38833.log.gz 2024-03-29T10:51:47.171 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-osd.2.log: /var/log/ceph/ceph-client.admin.35183.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.35183.log.gz 2024-03-29T10:51:47.171 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38480.log 2024-03-29T10:51:47.171 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.39379.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39730.log 2024-03-29T10:51:47.171 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.38480.log: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.39379.log.gz 2024-03-29T10:51:47.171 INFO:teuthology.orchestra.run.smithi088.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.38480.log.gz 2024-03-29T10:51:47.172 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40354.log 2024-03-29T10:51:47.172 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.39730.log: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.39730.log.gz 2024-03-29T10:51:47.172 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38558.log 2024-03-29T10:51:47.172 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.40354.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41740.log 2024-03-29T10:51:47.172 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.38558.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.40354.log.gz 2024-03-29T10:51:47.172 INFO:teuthology.orchestra.run.smithi088.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.38558.log.gz 2024-03-29T10:51:47.172 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39457.log 2024-03-29T10:51:47.172 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.41740.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40081.log 2024-03-29T10:51:47.172 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.39457.log: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.41740.log.gz 2024-03-29T10:51:47.172 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38872.log 2024-03-29T10:51:47.172 INFO:teuthology.orchestra.run.smithi088.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.39457.log.gz 2024-03-29T10:51:47.172 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.40081.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41178.log 2024-03-29T10:51:47.172 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.38872.log: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.40081.log.gz 2024-03-29T10:51:47.172 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38675.log 2024-03-29T10:51:47.172 INFO:teuthology.orchestra.run.smithi088.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.38872.log.gz/var/log/ceph/ceph-client.admin.41178.log: 2024-03-29T10:51:47.173 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41465.log 2024-03-29T10:51:47.173 INFO:teuthology.orchestra.run.smithi088.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.41178.log.gz 2024-03-29T10:51:47.173 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.38675.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41506.log 2024-03-29T10:51:47.173 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.41465.log: 85.6% -- replaced with /var/log/ceph/ceph-client.admin.38675.log.gz 2024-03-29T10:51:47.173 INFO:teuthology.orchestra.run.smithi088.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.41465.log.gz 2024-03-29T10:51:47.173 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41412.log 2024-03-29T10:51:47.173 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38755.log 2024-03-29T10:51:47.173 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.41506.log: /var/log/ceph/ceph-client.admin.41412.log: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.41506.log.gz 2024-03-29T10:51:47.173 INFO:teuthology.orchestra.run.smithi088.stderr: 85.6% -- replaced with /var/log/ceph/ceph-client.admin.41412.log.gz 2024-03-29T10:51:47.173 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38636.log 2024-03-29T10:51:47.173 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-03-29T10:51:47.173 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.38755.log: /var/log/ceph/ceph-client.admin.38636.log: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.38755.log.gz 2024-03-29T10:51:47.173 INFO:teuthology.orchestra.run.smithi088.stderr: 85.5% -- replaced with /var/log/ceph/ceph-client.admin.38636.log.gz 2024-03-29T10:51:47.180 INFO:teuthology.orchestra.run.smithi196.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-03-29T10:51:47.180 INFO:teuthology.orchestra.run.smithi196.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34788.log 2024-03-29T10:51:47.180 INFO:teuthology.orchestra.run.smithi196.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34868.log 2024-03-29T10:51:47.180 INFO:teuthology.orchestra.run.smithi196.stderr:/var/log/ceph/ceph-osd.6.log: /var/log/ceph/ceph-client.admin.34788.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34988.log 2024-03-29T10:51:47.180 INFO:teuthology.orchestra.run.smithi196.stderr:/var/log/ceph/ceph-client.admin.34868.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.34788.log.gz 2024-03-29T10:51:47.180 INFO:teuthology.orchestra.run.smithi196.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-29T10:51:47.180 INFO:teuthology.orchestra.run.smithi196.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-03-29T10:51:47.181 INFO:teuthology.orchestra.run.smithi196.stderr:/var/log/ceph/ceph-client.admin.34988.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.34868.log.gz 2024-03-29T10:51:47.181 INFO:teuthology.orchestra.run.smithi196.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34828.log 2024-03-29T10:51:47.181 INFO:teuthology.orchestra.run.smithi196.stderr:/var/log/ceph/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-03-29T10:51:47.181 INFO:teuthology.orchestra.run.smithi196.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.34988.log.gz 2024-03-29T10:51:47.181 INFO:teuthology.orchestra.run.smithi196.stderr: 88.1% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-29T10:51:47.181 INFO:teuthology.orchestra.run.smithi196.stderr:/var/log/ceph/ceph-client.admin.34828.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-03-29T10:51:47.181 INFO:teuthology.orchestra.run.smithi196.stderr:/var/log/ceph/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34748.log 2024-03-29T10:51:47.181 INFO:teuthology.orchestra.run.smithi196.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.34828.log.gz 2024-03-29T10:51:47.181 INFO:teuthology.orchestra.run.smithi196.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-03-29T10:51:47.181 INFO:teuthology.orchestra.run.smithi196.stderr:/var/log/ceph/ceph-client.admin.34748.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34948.log 2024-03-29T10:51:47.181 INFO:teuthology.orchestra.run.smithi196.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-03-29T10:51:47.181 INFO:teuthology.orchestra.run.smithi196.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.34748.log.gz 2024-03-29T10:51:47.181 INFO:teuthology.orchestra.run.smithi196.stderr:/var/log/ceph/ceph-client.admin.34948.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34708.log 2024-03-29T10:51:47.181 INFO:teuthology.orchestra.run.smithi196.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-03-29T10:51:47.181 INFO:teuthology.orchestra.run.smithi196.stderr: 87.8%/var/log/ceph/ceph-client.admin.34708.log: -- replaced with /var/log/ceph/ceph-client.admin.34948.log.gz 2024-03-29T10:51:47.182 INFO:teuthology.orchestra.run.smithi196.stderr:/var/log/ceph/ceph.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.34708.log.gz 2024-03-29T10:51:47.182 INFO:teuthology.orchestra.run.smithi196.stderr: 87.2% -- replaced with /var/log/ceph/ceph.log.gz 2024-03-29T10:51:47.182 INFO:teuthology.orchestra.run.smithi196.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34908.log 2024-03-29T10:51:47.182 INFO:teuthology.orchestra.run.smithi196.stderr:/var/log/ceph/ceph-mon.b.log: /var/log/ceph/ceph-client.admin.34908.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.34908.log.gz 2024-03-29T10:51:47.182 INFO:teuthology.orchestra.run.smithi196.stderr: 93.7% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-03-29T10:51:47.206 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-mon.c.log: 92.3% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2024-03-29T10:51:58.875 INFO:teuthology.orchestra.run.smithi088.stderr: 94.6% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-03-29T10:51:58.890 INFO:teuthology.orchestra.run.smithi088.stderr: 94.6% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-03-29T10:51:58.923 INFO:teuthology.orchestra.run.smithi196.stderr: 94.7% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-03-29T10:51:58.963 INFO:teuthology.orchestra.run.smithi196.stderr: 94.7% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-03-29T10:51:59.031 INFO:teuthology.orchestra.run.smithi196.stderr: 94.7% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-03-29T10:52:02.731 INFO:teuthology.orchestra.run.smithi088.stderr: 94.6% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-03-29T10:52:02.824 INFO:teuthology.orchestra.run.smithi196.stderr: 94.7% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-03-29T10:52:03.121 INFO:teuthology.orchestra.run.smithi088.stderr: 94.6% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-03-29T10:52:39.767 INFO:teuthology.orchestra.run.smithi088.stderr: 92.8% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-03-29T10:52:40.426 INFO:teuthology.orchestra.run.smithi088.stderr: 92.8% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-03-29T10:52:40.428 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-03-29T10:52:40.428 INFO:teuthology.orchestra.run.smithi088.stderr:real 0m53.323s 2024-03-29T10:52:40.428 INFO:teuthology.orchestra.run.smithi088.stderr:user 2m34.940s 2024-03-29T10:52:40.428 INFO:teuthology.orchestra.run.smithi088.stderr:sys 0m5.748s 2024-03-29T10:52:58.334 INFO:teuthology.orchestra.run.smithi196.stderr: 92.5% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-03-29T10:52:58.335 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-29T10:52:58.335 INFO:teuthology.orchestra.run.smithi196.stderr:real 1m11.187s 2024-03-29T10:52:58.335 INFO:teuthology.orchestra.run.smithi196.stderr:user 1m57.533s 2024-03-29T10:52:58.335 INFO:teuthology.orchestra.run.smithi196.stderr:sys 0m4.525s 2024-03-29T10:52:58.336 INFO:tasks.ceph:Archiving logs... 2024-03-29T10:52:58.336 DEBUG:teuthology.misc:Transferring archived files from smithi088:/var/log/ceph to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622648/remote/smithi088/log 2024-03-29T10:52:58.338 DEBUG:teuthology.orchestra.run.smithi088:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-29T10:53:22.257 DEBUG:teuthology.misc:Transferring archived files from smithi196:/var/log/ceph to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622648/remote/smithi196/log 2024-03-29T10:53:22.258 DEBUG:teuthology.orchestra.run.smithi196:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-29T10:53:42.182 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1913, in task with contextutil.nested(*subtasks): File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds 2024-03-29T10:53:42.387 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=26c647aa89ec4bafabe80d8af362a589 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1913, in task with contextutil.nested(*subtasks): File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds 2024-03-29T10:53:42.391 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-03-29T10:53:42.399 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-29T10:53:42.436 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-29T10:53:42.436 DEBUG:teuthology.orchestra.run.smithi088:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-29T10:53:42.450 DEBUG:teuthology.orchestra.run.smithi196:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-29T10:53:42.531 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-29T10:53:42.539 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-29T10:53:42.540 DEBUG:teuthology.orchestra.run.smithi088:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-29T10:53:42.543 DEBUG:teuthology.orchestra.run.smithi196:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-29T10:53:42.556 INFO:teuthology.orchestra.run.smithi088.stderr:bash: line 1: ntpq: command not found 2024-03-29T10:53:42.558 INFO:teuthology.orchestra.run.smithi196.stderr:bash: line 1: ntpq: command not found 2024-03-29T10:53:42.561 INFO:teuthology.orchestra.run.smithi088.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-29T10:53:42.561 INFO:teuthology.orchestra.run.smithi088.stdout:=============================================================================== 2024-03-29T10:53:42.561 INFO:teuthology.orchestra.run.smithi088.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 47 +415us[ +415us] +/- 72ms 2024-03-29T10:53:42.561 INFO:teuthology.orchestra.run.smithi088.stdout:^* hv02.front.sepia.ceph.com 3 6 377 48 -182us[ -180us] +/- 51ms 2024-03-29T10:53:42.562 INFO:teuthology.orchestra.run.smithi088.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 48 -191us[ -191us] +/- 82ms 2024-03-29T10:53:42.562 INFO:teuthology.orchestra.run.smithi088.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T10:53:42.562 INFO:teuthology.orchestra.run.smithi196.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-29T10:53:42.562 INFO:teuthology.orchestra.run.smithi196.stdout:=============================================================================== 2024-03-29T10:53:42.563 INFO:teuthology.orchestra.run.smithi196.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 50 +421us[ +421us] +/- 72ms 2024-03-29T10:53:42.563 INFO:teuthology.orchestra.run.smithi196.stdout:^* hv02.front.sepia.ceph.com 3 6 377 50 -185us[ -181us] +/- 51ms 2024-03-29T10:53:42.563 INFO:teuthology.orchestra.run.smithi196.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 49 -213us[ -213us] +/- 82ms 2024-03-29T10:53:42.563 INFO:teuthology.orchestra.run.smithi196.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T10:53:42.564 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-29T10:53:42.573 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-29T10:53:42.573 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-29T10:53:42.603 DEBUG:teuthology.orchestra.run.smithi088:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-29T10:53:42.672 DEBUG:teuthology.orchestra.run.smithi196:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-29T10:53:42.743 DEBUG:teuthology.orchestra.run.smithi088:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-29T10:53:42.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T10:53:42.774 DEBUG:teuthology.orchestra.run.smithi196:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-29T10:53:42.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T10:53:42.805 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-29T10:53:42.814 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-29T10:53:42.837 INFO:teuthology.task.internal:Duration was 1213.381576 seconds 2024-03-29T10:53:42.837 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-29T10:53:42.870 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-29T10:53:42.870 DEBUG:teuthology.orchestra.run.smithi088:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-29T10:53:42.873 DEBUG:teuthology.orchestra.run.smithi196:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-29T10:53:42.914 INFO:teuthology.orchestra.run.smithi088.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-29T10:53:42.917 INFO:teuthology.orchestra.run.smithi196.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-29T10:53:43.189 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-29T10:53:43.189 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi088.front.sepia.ceph.com 2024-03-29T10:53:43.189 DEBUG:teuthology.orchestra.run.smithi088:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-29T10:53:43.230 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi196.front.sepia.ceph.com 2024-03-29T10:53:43.230 DEBUG:teuthology.orchestra.run.smithi196:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-29T10:53:43.252 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-29T10:53:43.252 DEBUG:teuthology.orchestra.run.smithi088:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-29T10:53:43.272 DEBUG:teuthology.orchestra.run.smithi196:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-29T10:53:43.320 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-29T10:53:43.320 DEBUG:teuthology.orchestra.run.smithi088:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-29T10:53:43.343 DEBUG:teuthology.orchestra.run.smithi196:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-29T10:53:43.455 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-29T10:53:43.465 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-29T10:53:43.465 DEBUG:teuthology.orchestra.run.smithi088:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-29T10:53:43.492 DEBUG:teuthology.orchestra.run.smithi196:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-29T10:53:43.521 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-29T10:53:43.530 DEBUG:teuthology.orchestra.run.smithi088:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-29T10:53:43.534 DEBUG:teuthology.orchestra.run.smithi196:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-29T10:53:43.557 INFO:teuthology.orchestra.run.smithi088.stdout:kernel.core_pattern = core 2024-03-29T10:53:43.586 INFO:teuthology.orchestra.run.smithi196.stdout:kernel.core_pattern = core 2024-03-29T10:53:43.621 DEBUG:teuthology.orchestra.run.smithi088:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-29T10:53:43.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T10:53:43.638 DEBUG:teuthology.orchestra.run.smithi196:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-29T10:53:43.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T10:53:43.676 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-29T10:53:43.687 INFO:teuthology.task.internal:Transferring archived files... 2024-03-29T10:53:43.688 DEBUG:teuthology.misc:Transferring archived files from smithi088:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622648/remote/smithi088 2024-03-29T10:53:43.688 DEBUG:teuthology.orchestra.run.smithi088:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-29T10:53:43.725 DEBUG:teuthology.misc:Transferring archived files from smithi196:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622648/remote/smithi196 2024-03-29T10:53:43.725 DEBUG:teuthology.orchestra.run.smithi196:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-29T10:53:43.762 INFO:teuthology.task.internal:Removing archive directory... 2024-03-29T10:53:43.762 DEBUG:teuthology.orchestra.run.smithi088:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-29T10:53:43.764 DEBUG:teuthology.orchestra.run.smithi196:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-29T10:53:43.815 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-29T10:53:43.826 INFO:teuthology.task.internal:Not uploading archives. 2024-03-29T10:53:43.826 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-29T10:53:43.854 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-29T10:53:43.854 DEBUG:teuthology.orchestra.run.smithi088:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-29T10:53:43.856 DEBUG:teuthology.orchestra.run.smithi196:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-29T10:53:43.868 INFO:teuthology.orchestra.run.smithi088.stdout: 265305 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 29 10:53 /home/ubuntu/cephtest 2024-03-29T10:53:43.870 INFO:teuthology.orchestra.run.smithi196.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 29 10:53 /home/ubuntu/cephtest 2024-03-29T10:53:43.872 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-29T10:53:43.888 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-29T10:53:43.963 INFO:teuthology.nuke:Checking targets against current locks 2024-03-29T10:53:43.990 DEBUG:teuthology.nuke:shortname: smithi088 2024-03-29T10:53:43.990 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-29T10:53:44.017 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi088.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622648', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-29 10:24:34.037213', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJC0hr90S7oLfAoelGBHu4/0U2rkf61hfisamgYRLn7TC82DQRaG4HpL3kXnRcujlwQVB8jMOFZCfEWpvvZDsdI='} 2024-03-29T10:53:44.020 DEBUG:teuthology.nuke:shortname: smithi196 2024-03-29T10:53:44.021 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-29T10:53:44.045 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi196.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622648', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-29 10:24:34.035537', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOmc8TxTRpU1Lg3jXUaU3i4KUHAnuTYSGL8ZhlCPHEUiKLNgxAitJvjwBVcGhpvmK8w1uOgNB2wJREwIlBz5vvs='} 2024-03-29T10:53:44.077 INFO:teuthology.orchestra.console.smithi088:Power off 2024-03-29T10:53:44.077 DEBUG:teuthology.orchestra.console.smithi088:pexpect command: ipmitool -H smithi088.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-29T10:53:44.104 DEBUG:teuthology.orchestra.console.smithi088:power off output: Chassis Power Control: Down/Off 2024-03-29T10:53:44.105 DEBUG:teuthology.orchestra.console.smithi088:pexpect command: ipmitool -H smithi088.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T10:53:44.121 INFO:teuthology.orchestra.console.smithi196:Power off 2024-03-29T10:53:44.121 DEBUG:teuthology.orchestra.console.smithi196:pexpect command: ipmitool -H smithi196.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-29T10:53:44.130 DEBUG:teuthology.orchestra.console.smithi088:check power output: Chassis Power is on 2024-03-29T10:53:44.145 DEBUG:teuthology.orchestra.console.smithi196:power off output: Chassis Power Control: Down/Off 2024-03-29T10:53:44.146 DEBUG:teuthology.orchestra.console.smithi196:pexpect command: ipmitool -H smithi196.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T10:53:44.171 DEBUG:teuthology.orchestra.console.smithi196:check power output: Chassis Power is on 2024-03-29T10:53:48.132 DEBUG:teuthology.orchestra.console.smithi088:pexpect command: ipmitool -H smithi088.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T10:53:48.172 DEBUG:teuthology.orchestra.console.smithi196:pexpect command: ipmitool -H smithi196.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T10:53:48.242 DEBUG:teuthology.orchestra.console.smithi088:check power output: Chassis Power is on 2024-03-29T10:53:48.282 DEBUG:teuthology.orchestra.console.smithi196:check power output: Chassis Power is on 2024-03-29T10:53:52.246 DEBUG:teuthology.orchestra.console.smithi088:pexpect command: ipmitool -H smithi088.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T10:53:52.283 DEBUG:teuthology.orchestra.console.smithi196:pexpect command: ipmitool -H smithi196.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T10:53:52.358 DEBUG:teuthology.orchestra.console.smithi088:check power output: Chassis Power is on 2024-03-29T10:53:52.394 DEBUG:teuthology.orchestra.console.smithi196:check power output: Chassis Power is off 2024-03-29T10:53:52.495 INFO:teuthology.orchestra.console.smithi196:Power off completed 2024-03-29T10:53:56.362 DEBUG:teuthology.orchestra.console.smithi088:pexpect command: ipmitool -H smithi088.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T10:53:56.473 DEBUG:teuthology.orchestra.console.smithi088:check power output: Chassis Power is off 2024-03-29T10:53:56.574 INFO:teuthology.orchestra.console.smithi088:Power off completed 2024-03-29T10:53:56.678 INFO:teuthology.run:Summary data: description: rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zstd rados tasks/rados_api_tests validater/lockdep} duration: 1213.3815755844116 failure_reason: reached maximum tries (91) after waiting for 540 seconds flavor: default owner: scheduled_amathuri@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=26c647aa89ec4bafabe80d8af362a589 status: fail success: false 2024-03-29T10:53:56.679 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-29T10:53:56.768 INFO:teuthology.run:FAIL